2024-06-27T21:19:16.680 INFO:root:teuthology version: 0.0.1.dev329+g544fecb 2024-06-27T21:19:16.680 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775602 --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} --verbose --name yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775602/orig.config.yaml 2024-06-27T21:19:16.703 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-27T21:19:16.788 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775602 branch: wip-yuri12-testing-2024-06-26-0904 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} email: yweinste@redhat.com first_in_suite: false job_id: '7775602' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri12-testing-2024-06-26-0904 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: f51acc18cc950586c82c58d666661d4391fc1122 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f51acc18cc950586c82c58d666661d4391fc1122 workunit: branch: wip-yuri12-testing-2024-06-26-0904 sha1: f51acc18cc950586c82c58d666661d4391fc1122 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9950 sha1: f51acc18cc950586c82c58d666661d4391fc1122 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri12-testing-2024-06-26-0904 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f51acc18cc950586c82c58d666661d4391fc1122 targets: smithi042.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmcHDQzTPnnSI+TUZRT2NWPqqy9dtHv6lQFn78b9byPpL8a30fAWk+MkgGHSRH29vonl38VFbTqb0trtFJbSkQ= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 544fecbcd55f3d2b6f478823823ce40cbefef1d4 timestamp: 2024-06-27_14:07:21 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698093 2024-06-27T21:19:16.788 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa; will attempt to use it 2024-06-27T21:19:16.789 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa/tasks 2024-06-27T21:19:16.789 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-27T21:19:16.791 INFO:teuthology.task.internal:Checking packages... 2024-06-27T21:19:16.811 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f51acc18cc950586c82c58d666661d4391fc1122' 2024-06-27T21:19:16.812 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-27T21:19:16.812 INFO:teuthology.packaging:ref: None 2024-06-27T21:19:16.812 INFO:teuthology.packaging:tag: None 2024-06-27T21:19:16.812 INFO:teuthology.packaging:branch: wip-yuri12-testing-2024-06-26-0904 2024-06-27T21:19:16.812 INFO:teuthology.packaging:sha1: f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:19:16.812 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri12-testing-2024-06-26-0904 2024-06-27T21:19:17.117 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-4578-gf51acc18-1jammy 2024-06-27T21:19:17.119 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-27T21:19:17.126 INFO:teuthology.task.internal:no buildpackages task found 2024-06-27T21:19:17.126 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-27T21:19:17.139 INFO:teuthology.task.internal:Saving configuration 2024-06-27T21:19:17.148 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-27T21:19:17.155 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-27T21:19:17.180 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775602', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-27 21:13:34.031904', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID3X2zyD35fC/CaKmrepFiVGHRrIsnxXlhKfWxWhSyzC'} 2024-06-27T21:19:17.180 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-27T21:19:17.186 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-06-27T21:19:17.187 INFO:teuthology.run_tasks:Running task console_log... 2024-06-27T21:19:17.223 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f07f29cfbe0>, signals=[15]) 2024-06-27T21:19:17.223 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-27T21:19:17.229 INFO:teuthology.task.internal:Opening connections... 2024-06-27T21:19:17.229 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi042.front.sepia.ceph.com 2024-06-27T21:19:17.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T21:19:17.315 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-27T21:19:17.322 DEBUG:teuthology.orchestra.run.smithi042:> uname -m 2024-06-27T21:19:17.423 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2024-06-27T21:19:17.423 DEBUG:teuthology.orchestra.run.smithi042:> cat /etc/os-release 2024-06-27T21:19:17.469 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-06-27T21:19:17.469 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="Ubuntu" 2024-06-27T21:19:17.469 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="22.04" 2024-06-27T21:19:17.470 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-06-27T21:19:17.470 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_CODENAME=jammy 2024-06-27T21:19:17.470 INFO:teuthology.orchestra.run.smithi042.stdout:ID=ubuntu 2024-06-27T21:19:17.470 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE=debian 2024-06-27T21:19:17.470 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://www.ubuntu.com/" 2024-06-27T21:19:17.470 INFO:teuthology.orchestra.run.smithi042.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-06-27T21:19:17.470 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-06-27T21:19:17.470 INFO:teuthology.orchestra.run.smithi042.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-06-27T21:19:17.470 INFO:teuthology.orchestra.run.smithi042.stdout:UBUNTU_CODENAME=jammy 2024-06-27T21:19:17.471 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2024-06-27T21:19:17.494 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-27T21:19:17.502 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-27T21:19:17.513 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-27T21:19:17.513 DEBUG:teuthology.orchestra.run.smithi042:> test '!' -e /home/ubuntu/cephtest 2024-06-27T21:19:17.518 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-27T21:19:17.525 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-27T21:19:17.525 DEBUG:teuthology.orchestra.run.smithi042:> test -z $(ls -A /var/lib/ceph) 2024-06-27T21:19:17.576 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-27T21:19:17.606 INFO:teuthology.run_tasks:Running task kernel... 2024-06-27T21:19:17.620 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-27T21:19:17.621 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-27T21:19:17.621 DEBUG:teuthology.orchestra.run.smithi042:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-27T21:19:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:17.626 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-06-27T21:19:17.673 INFO:teuthology.orchestra.run.smithi042.stdout:5.15.0-56-generic 2024-06-27T21:19:17.673 INFO:teuthology.task.kernel:Running kernel on smithi042: 5.15.0-56-generic 2024-06-27T21:19:17.673 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get clean 2024-06-27T21:19:17.799 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get update 2024-06-27T21:19:17.993 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-27T21:19:18.021 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-06-27T21:19:18.049 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-06-27T21:19:18.155 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-06-27T21:19:18.241 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,774 kB] 2024-06-27T21:19:18.396 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [649 kB] 2024-06-27T21:19:18.407 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [323 kB] 2024-06-27T21:19:18.412 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-06-27T21:19:18.413 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.9 kB] 2024-06-27T21:19:18.413 INFO:teuthology.orchestra.run.smithi042.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,063 kB] 2024-06-27T21:19:18.458 INFO:teuthology.orchestra.run.smithi042.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [351 kB] 2024-06-27T21:19:18.460 INFO:teuthology.orchestra.run.smithi042.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-06-27T21:19:18.461 INFO:teuthology.orchestra.run.smithi042.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,090 kB] 2024-06-27T21:19:18.477 INFO:teuthology.orchestra.run.smithi042.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [710 kB] 2024-06-27T21:19:18.488 INFO:teuthology.orchestra.run.smithi042.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [492 kB] 2024-06-27T21:19:18.491 INFO:teuthology.orchestra.run.smithi042.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [252 kB] 2024-06-27T21:19:18.494 INFO:teuthology.orchestra.run.smithi042.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-06-27T21:19:18.495 INFO:teuthology.orchestra.run.smithi042.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-06-27T21:19:18.496 INFO:teuthology.orchestra.run.smithi042.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,744 B] 2024-06-27T21:19:18.506 INFO:teuthology.orchestra.run.smithi042.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-06-27T21:19:18.507 INFO:teuthology.orchestra.run.smithi042.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-06-27T21:19:18.508 INFO:teuthology.orchestra.run.smithi042.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-06-27T21:19:18.509 INFO:teuthology.orchestra.run.smithi042.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-06-27T21:19:18.510 INFO:teuthology.orchestra.run.smithi042.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-06-27T21:19:18.528 INFO:teuthology.orchestra.run.smithi042.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-06-27T21:19:18.528 INFO:teuthology.orchestra.run.smithi042.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-06-27T21:19:18.528 INFO:teuthology.orchestra.run.smithi042.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.6 kB] 2024-06-27T21:19:18.555 INFO:teuthology.orchestra.run.smithi042.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-06-27T21:19:18.555 INFO:teuthology.orchestra.run.smithi042.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-06-27T21:19:18.764 INFO:teuthology.orchestra.run.smithi042.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,564 kB] 2024-06-27T21:19:18.933 INFO:teuthology.orchestra.run.smithi042.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [265 kB] 2024-06-27T21:19:18.945 INFO:teuthology.orchestra.run.smithi042.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-06-27T21:19:18.945 INFO:teuthology.orchestra.run.smithi042.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,004 kB] 2024-06-27T21:19:19.028 INFO:teuthology.orchestra.run.smithi042.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.3 kB] 2024-06-27T21:19:19.029 INFO:teuthology.orchestra.run.smithi042.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [342 kB] 2024-06-27T21:19:19.037 INFO:teuthology.orchestra.run.smithi042.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-06-27T21:19:19.037 INFO:teuthology.orchestra.run.smithi042.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [612 kB] 2024-06-27T21:19:19.051 INFO:teuthology.orchestra.run.smithi042.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [875 kB] 2024-06-27T21:19:19.069 INFO:teuthology.orchestra.run.smithi042.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [170 kB] 2024-06-27T21:19:19.086 INFO:teuthology.orchestra.run.smithi042.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-06-27T21:19:19.097 INFO:teuthology.orchestra.run.smithi042.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-06-27T21:19:19.097 INFO:teuthology.orchestra.run.smithi042.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-06-27T21:19:19.178 INFO:teuthology.orchestra.run.smithi042.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-06-27T21:19:19.178 INFO:teuthology.orchestra.run.smithi042.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-06-27T21:19:24.793 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 14.4 MB in 2s (6,665 kB/s) 2024-06-27T21:19:26.274 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-06-27T21:19:26.291 DEBUG:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-27T21:19:26.346 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-06-27T21:19:26.532 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2024-06-27T21:19:26.532 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2024-06-27T21:19:26.699 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2024-06-27T21:19:26.699 INFO:teuthology.orchestra.run.smithi042.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-27T21:19:26.699 INFO:teuthology.orchestra.run.smithi042.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-27T21:19:26.700 INFO:teuthology.orchestra.run.smithi042.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-27T21:19:26.700 INFO:teuthology.orchestra.run.smithi042.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-06-27T21:19:26.700 INFO:teuthology.orchestra.run.smithi042.stdout: python2.7-minimal 2024-06-27T21:19:26.700 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-27T21:19:26.701 INFO:teuthology.orchestra.run.smithi042.stdout:The following additional packages will be installed: 2024-06-27T21:19:26.701 INFO:teuthology.orchestra.run.smithi042.stdout: linux-generic linux-headers-5.15.0-113 linux-headers-5.15.0-113-generic 2024-06-27T21:19:26.701 INFO:teuthology.orchestra.run.smithi042.stdout: linux-headers-generic linux-image-5.15.0-113-generic 2024-06-27T21:19:26.701 INFO:teuthology.orchestra.run.smithi042.stdout: linux-modules-5.15.0-113-generic linux-modules-extra-5.15.0-113-generic 2024-06-27T21:19:26.703 INFO:teuthology.orchestra.run.smithi042.stdout:Suggested packages: 2024-06-27T21:19:26.703 INFO:teuthology.orchestra.run.smithi042.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-06-27T21:19:26.703 INFO:teuthology.orchestra.run.smithi042.stdout:Recommended packages: 2024-06-27T21:19:26.703 INFO:teuthology.orchestra.run.smithi042.stdout: thermald 2024-06-27T21:19:26.749 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2024-06-27T21:19:26.749 INFO:teuthology.orchestra.run.smithi042.stdout: linux-headers-5.15.0-113 linux-headers-5.15.0-113-generic 2024-06-27T21:19:26.749 INFO:teuthology.orchestra.run.smithi042.stdout: linux-image-5.15.0-113-generic linux-modules-5.15.0-113-generic 2024-06-27T21:19:26.749 INFO:teuthology.orchestra.run.smithi042.stdout: linux-modules-extra-5.15.0-113-generic 2024-06-27T21:19:26.750 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be upgraded: 2024-06-27T21:19:26.751 INFO:teuthology.orchestra.run.smithi042.stdout: linux-generic linux-headers-generic linux-image-generic 2024-06-27T21:19:26.867 INFO:teuthology.orchestra.run.smithi042.stdout:3 upgraded, 5 newly installed, 0 to remove and 328 not upgraded. 2024-06-27T21:19:26.867 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 113 MB of archives. 2024-06-27T21:19:26.867 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 583 MB of additional disk space will be used. 2024-06-27T21:19:26.867 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-113-generic amd64 5.15.0-113.123 [22.6 MB] 2024-06-27T21:19:27.690 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-113-generic amd64 5.15.0-113.123 [11.5 MB] 2024-06-27T21:19:27.875 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-113-generic amd64 5.15.0-113.123 [63.9 MB] 2024-06-27T21:19:29.163 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.113.113 [1,692 B] 2024-06-27T21:19:29.163 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.113.113 [2,488 B] 2024-06-27T21:19:29.163 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-113 all 5.15.0-113.123 [12.3 MB] 2024-06-27T21:19:29.358 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-113-generic amd64 5.15.0-113.123 [2,895 kB] 2024-06-27T21:19:29.389 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.113.113 [2,346 B] 2024-06-27T21:19:29.797 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 113 MB in 3s (43.1 MB/s) 2024-06-27T21:19:29.970 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-modules-5.15.0-113-generic. 2024-06-27T21:19:32.153 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-06-27T21:19:32.158 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../0-linux-modules-5.15.0-113-generic_5.15.0-113.123_amd64.deb ... 2024-06-27T21:19:32.386 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-modules-5.15.0-113-generic (5.15.0-113.123) ... 2024-06-27T21:19:34.485 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-image-5.15.0-113-generic. 2024-06-27T21:19:34.507 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../1-linux-image-5.15.0-113-generic_5.15.0-113.123_amd64.deb ... 2024-06-27T21:19:34.600 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-image-5.15.0-113-generic (5.15.0-113.123) ... 2024-06-27T21:19:35.063 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-113-generic. 2024-06-27T21:19:35.085 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-113-generic_5.15.0-113.123_amd64.deb ... 2024-06-27T21:19:35.121 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-modules-extra-5.15.0-113-generic (5.15.0-113.123) ... 2024-06-27T21:19:41.522 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../3-linux-generic_5.15.0.113.113_amd64.deb ... 2024-06-27T21:19:41.648 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-generic (5.15.0.113.113) over (5.15.0.56.54) ... 2024-06-27T21:19:42.107 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.113.113_amd64.deb ... 2024-06-27T21:19:42.446 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-image-generic (5.15.0.113.113) over (5.15.0.56.54) ... 2024-06-27T21:19:42.721 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-headers-5.15.0-113. 2024-06-27T21:19:42.744 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../5-linux-headers-5.15.0-113_5.15.0-113.123_all.deb ... 2024-06-27T21:19:42.779 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-5.15.0-113 (5.15.0-113.123) ... 2024-06-27T21:19:48.174 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-headers-5.15.0-113-generic. 2024-06-27T21:19:48.212 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../6-linux-headers-5.15.0-113-generic_5.15.0-113.123_amd64.deb ... 2024-06-27T21:19:48.240 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-5.15.0-113-generic (5.15.0-113.123) ... 2024-06-27T21:19:50.887 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.113.113_amd64.deb ... 2024-06-27T21:19:51.010 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-generic (5.15.0.113.113) over (5.15.0.56.54) ... 2024-06-27T21:19:51.302 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-5.15.0-113 (5.15.0-113.123) ... 2024-06-27T21:19:51.424 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-5.15.0-113-generic (5.15.0-113.123) ... 2024-06-27T21:19:51.550 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-generic (5.15.0.113.113) ... 2024-06-27T21:19:51.677 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-modules-5.15.0-113-generic (5.15.0-113.123) ... 2024-06-27T21:19:53.114 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-image-5.15.0-113-generic (5.15.0-113.123) ... 2024-06-27T21:19:54.664 INFO:teuthology.orchestra.run.smithi042.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-06-27T21:19:54.664 INFO:teuthology.orchestra.run.smithi042.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-06-27T21:19:54.664 INFO:teuthology.orchestra.run.smithi042.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-113-generic 2024-06-27T21:19:54.664 INFO:teuthology.orchestra.run.smithi042.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-113-generic 2024-06-27T21:19:54.808 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-modules-extra-5.15.0-113-generic (5.15.0-113.123) ... 2024-06-27T21:19:56.250 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-image-generic (5.15.0.113.113) ... 2024-06-27T21:19:56.376 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-generic (5.15.0.113.113) ... 2024-06-27T21:19:56.502 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for linux-image-5.15.0-113-generic (5.15.0-113.123) ... 2024-06-27T21:19:56.552 INFO:teuthology.orchestra.run.smithi042.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-06-27T21:19:56.552 INFO:teuthology.orchestra.run.smithi042.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-113-generic 2024-06-27T21:20:10.561 INFO:teuthology.orchestra.run.smithi042.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-06-27T21:20:10.561 INFO:teuthology.orchestra.run.smithi042.stdout:Sourcing file `/etc/default/grub' 2024-06-27T21:20:10.570 INFO:teuthology.orchestra.run.smithi042.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-27T21:20:10.599 INFO:teuthology.orchestra.run.smithi042.stdout:Generating grub configuration file ... 2024-06-27T21:20:10.864 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-5.15.0-113-generic 2024-06-27T21:20:10.885 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-5.15.0-113-generic 2024-06-27T21:20:11.183 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-27T21:20:11.189 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-27T21:20:11.261 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-27T21:20:11.267 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-27T21:20:11.437 INFO:teuthology.orchestra.run.smithi042.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-27T21:20:11.437 INFO:teuthology.orchestra.run.smithi042.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-06-27T21:20:11.437 INFO:teuthology.orchestra.run.smithi042.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-27T21:20:11.481 INFO:teuthology.orchestra.run.smithi042.stdout:done 2024-06-27T21:20:11.966 DEBUG:teuthology.orchestra.run.smithi042:> dpkg -s linux-image-generic 2024-06-27T21:20:11.990 INFO:teuthology.orchestra.run.smithi042.stdout:Package: linux-image-generic 2024-06-27T21:20:11.990 INFO:teuthology.orchestra.run.smithi042.stdout:Status: install ok installed 2024-06-27T21:20:11.990 INFO:teuthology.orchestra.run.smithi042.stdout:Priority: optional 2024-06-27T21:20:11.990 INFO:teuthology.orchestra.run.smithi042.stdout:Section: kernel 2024-06-27T21:20:11.990 INFO:teuthology.orchestra.run.smithi042.stdout:Installed-Size: 21 2024-06-27T21:20:11.990 INFO:teuthology.orchestra.run.smithi042.stdout:Maintainer: Ubuntu Kernel Team 2024-06-27T21:20:11.990 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: amd64 2024-06-27T21:20:11.990 INFO:teuthology.orchestra.run.smithi042.stdout:Source: linux-meta 2024-06-27T21:20:11.991 INFO:teuthology.orchestra.run.smithi042.stdout:Version: 5.15.0.113.113 2024-06-27T21:20:11.991 INFO:teuthology.orchestra.run.smithi042.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-113), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-27T21:20:11.991 INFO:teuthology.orchestra.run.smithi042.stdout:Depends: linux-image-5.15.0-113-generic, linux-modules-extra-5.15.0-113-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-27T21:20:11.991 INFO:teuthology.orchestra.run.smithi042.stdout:Recommends: thermald 2024-06-27T21:20:11.991 INFO:teuthology.orchestra.run.smithi042.stdout:Description: Generic Linux kernel image 2024-06-27T21:20:11.991 INFO:teuthology.orchestra.run.smithi042.stdout: This package will always depend on the latest generic kernel image 2024-06-27T21:20:11.991 INFO:teuthology.orchestra.run.smithi042.stdout: available. 2024-06-27T21:20:11.991 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-113-generic 2024-06-27T21:20:11.991 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-27T21:20:11.991 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-27T21:20:11.992 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-27T21:20:11.992 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi042.front.sepia.ceph.com, path=None, version=distro) 2024-06-27T21:20:11.992 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get clean 2024-06-27T21:20:12.160 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get update 2024-06-27T21:20:12.424 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-27T21:20:12.437 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-27T21:20:12.494 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-06-27T21:20:12.577 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-27T21:20:13.903 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-06-27T21:20:13.921 DEBUG:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-27T21:20:13.986 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-06-27T21:20:14.195 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2024-06-27T21:20:14.196 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2024-06-27T21:20:14.382 INFO:teuthology.orchestra.run.smithi042.stdout:linux-image-generic is already the newest version (5.15.0.113.113). 2024-06-27T21:20:14.382 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2024-06-27T21:20:14.382 INFO:teuthology.orchestra.run.smithi042.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-27T21:20:14.382 INFO:teuthology.orchestra.run.smithi042.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-27T21:20:14.382 INFO:teuthology.orchestra.run.smithi042.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-27T21:20:14.383 INFO:teuthology.orchestra.run.smithi042.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-27T21:20:14.383 INFO:teuthology.orchestra.run.smithi042.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-27T21:20:14.383 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-27T21:20:14.428 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 328 not upgraded. 2024-06-27T21:20:14.430 DEBUG:teuthology.orchestra.run.smithi042:> dpkg -s linux-image-generic 2024-06-27T21:20:14.446 INFO:teuthology.orchestra.run.smithi042.stdout:Package: linux-image-generic 2024-06-27T21:20:14.446 INFO:teuthology.orchestra.run.smithi042.stdout:Status: install ok installed 2024-06-27T21:20:14.446 INFO:teuthology.orchestra.run.smithi042.stdout:Priority: optional 2024-06-27T21:20:14.446 INFO:teuthology.orchestra.run.smithi042.stdout:Section: kernel 2024-06-27T21:20:14.446 INFO:teuthology.orchestra.run.smithi042.stdout:Installed-Size: 21 2024-06-27T21:20:14.446 INFO:teuthology.orchestra.run.smithi042.stdout:Maintainer: Ubuntu Kernel Team 2024-06-27T21:20:14.446 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: amd64 2024-06-27T21:20:14.446 INFO:teuthology.orchestra.run.smithi042.stdout:Source: linux-meta 2024-06-27T21:20:14.446 INFO:teuthology.orchestra.run.smithi042.stdout:Version: 5.15.0.113.113 2024-06-27T21:20:14.446 INFO:teuthology.orchestra.run.smithi042.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-113), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-27T21:20:14.447 INFO:teuthology.orchestra.run.smithi042.stdout:Depends: linux-image-5.15.0-113-generic, linux-modules-extra-5.15.0-113-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-27T21:20:14.447 INFO:teuthology.orchestra.run.smithi042.stdout:Recommends: thermald 2024-06-27T21:20:14.447 INFO:teuthology.orchestra.run.smithi042.stdout:Description: Generic Linux kernel image 2024-06-27T21:20:14.447 INFO:teuthology.orchestra.run.smithi042.stdout: This package will always depend on the latest generic kernel image 2024-06-27T21:20:14.447 INFO:teuthology.orchestra.run.smithi042.stdout: available. 2024-06-27T21:20:14.447 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-06-27T21:20:14.494 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.kPAsyPYQLX 2024-06-27T21:20:14.494 DEBUG:teuthology.orchestra.run.smithi042:> sudo cp /boot/grub/grub.cfg /tmp/tmp.kPAsyPYQLX 2024-06-27T21:20:14.550 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.kPAsyPYQLX 2024-06-27T21:20:14.668 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.kPAsyPYQLX is 10KB 2024-06-27T21:20:14.683 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.kPAsyPYQLX 2024-06-27T21:20:14.688 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-06-27T21:20:14.742 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-06-27T21:20:14.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-06-27T21:20:14.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-06-27T21:20:14.810 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-113-generic 2024-06-27T21:20:14.810 DEBUG:teuthology.orchestra.run.smithi042:> sudo update-grub 2024-06-27T21:20:16.008 INFO:teuthology.orchestra.run.smithi042.stderr:Sourcing file `/etc/default/grub' 2024-06-27T21:20:16.010 INFO:teuthology.orchestra.run.smithi042.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-27T21:20:16.012 INFO:teuthology.orchestra.run.smithi042.stderr:Generating grub configuration file ... 2024-06-27T21:20:16.226 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-5.15.0-113-generic 2024-06-27T21:20:16.232 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-5.15.0-113-generic 2024-06-27T21:20:16.479 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-27T21:20:16.484 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-27T21:20:16.530 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-27T21:20:16.535 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-27T21:20:16.685 INFO:teuthology.orchestra.run.smithi042.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-27T21:20:16.685 INFO:teuthology.orchestra.run.smithi042.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-06-27T21:20:16.685 INFO:teuthology.orchestra.run.smithi042.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-27T21:20:16.707 INFO:teuthology.orchestra.run.smithi042.stderr:done 2024-06-27T21:20:16.717 DEBUG:teuthology.orchestra.run.smithi042:> sudo shutdown -r now 2024-06-27T21:20:46.750 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-27T21:20:46.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-06-27T21:20:46.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T21:21:05.315 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.42 2024-06-27T21:21:14.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-06-27T21:21:14.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T21:22:14.383 DEBUG:teuthology.orchestra.remote:timed out 2024-06-27T21:22:26.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-06-27T21:22:26.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T21:22:26.683 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-06-27T21:22:27.282 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-06-27T21:22:27.282 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-113-generic"... 2024-06-27T21:22:27.283 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-06-27T21:22:27.329 INFO:teuthology.orchestra.run.smithi042.stdout:5.15.0-113-generic 2024-06-27T21:22:27.329 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-113-generic vs 5.15.0-113-generic 2024-06-27T21:22:27.329 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-27T21:22:27.329 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-06-27T21:22:28.330 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-27T21:22:28.330 DEBUG:teuthology.orchestra.run.smithi042:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-27T21:22:28.406 INFO:teuthology.orchestra.run.smithi042.stdout:ttyS1 2024-06-27T21:22:28.426 DEBUG:teuthology.parallel:result is None 2024-06-27T21:22:28.427 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-27T21:22:28.434 INFO:teuthology.task.internal:Creating test directory... 2024-06-27T21:22:28.435 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-27T21:22:28.440 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-27T21:22:28.447 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-27T21:22:28.457 INFO:teuthology.task.internal:Creating archive directory... 2024-06-27T21:22:28.457 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-27T21:22:28.503 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-27T21:22:28.509 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-27T21:22:28.509 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-27T21:22:28.565 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:22:28.573 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:22:28.575 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-27T21:22:28.582 INFO:teuthology.task.internal:Configuring sudo... 2024-06-27T21:22:28.582 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-27T21:22:28.635 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-27T21:22:28.645 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-27T21:22:28.645 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-27T21:22:28.681 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-27T21:22:28.729 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-27T21:22:28.777 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-06-27T21:22:28.778 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-27T21:22:28.838 DEBUG:teuthology.orchestra.run.smithi042:> sudo service rsyslog restart 2024-06-27T21:22:28.916 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-27T21:22:28.924 INFO:teuthology.task.internal:Starting timer... 2024-06-27T21:22:28.924 INFO:teuthology.run_tasks:Running task pcp... 2024-06-27T21:22:28.935 INFO:teuthology.run_tasks:Running task selinux... 2024-06-27T21:22:28.950 DEBUG:teuthology.task.selinux:Excluding smithi042: OS 'ubuntu' does not support SELinux 2024-06-27T21:22:28.951 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-27T21:22:28.951 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-06-27T21:22:28.951 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-27T21:22:28.951 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-27T21:22:28.968 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-27T21:22:28.980 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-27T21:22:28.981 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi042.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-27T21:29:33.532 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi042.front.sepia.ceph.com')] 2024-06-27T21:29:33.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-06-27T21:29:33.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T21:29:33.617 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-06-27T21:29:33.698 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-06-27T21:29:33.698 INFO:teuthology.run_tasks:Running task clock... 2024-06-27T21:29:33.709 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-27T21:29:33.710 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-27T21:29:33.710 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-27T21:29:33.766 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-06-27T21:29:33.767 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: Command line: ntpd -gq 2024-06-27T21:29:33.767 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: ---------------------------------------------------- 2024-06-27T21:29:33.767 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: ntp-4 is maintained by Network Time Foundation, 2024-06-27T21:29:33.767 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-06-27T21:29:33.767 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: corporation. Support and training for ntp-4 are 2024-06-27T21:29:33.767 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: available at https://www.nwtime.org/support 2024-06-27T21:29:33.767 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: ---------------------------------------------------- 2024-06-27T21:29:33.768 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: proto: precision = 0.051 usec (-24) 2024-06-27T21:29:33.768 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: basedate set to 2022-02-04 2024-06-27T21:29:33.768 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: gps base set to 2022-02-06 (week 2196) 2024-06-27T21:29:33.768 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-27T21:29:33.768 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: restrict ::: KOD does nothing without LIMITED. 2024-06-27T21:29:33.768 INFO:teuthology.orchestra.run.smithi042.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-27T21:29:33.768 INFO:teuthology.orchestra.run.smithi042.stderr:restrict ::: KOD does nothing without LIMITED. 2024-06-27T21:29:33.769 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: Listen and drop on 0 v6wildcard [::]:123 2024-06-27T21:29:33.769 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-06-27T21:29:33.769 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: Listen normally on 2 lo 127.0.0.1:123 2024-06-27T21:29:33.770 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: Listen normally on 3 ens1f0 172.21.15.42:123 2024-06-27T21:29:33.770 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: Listen normally on 4 lo [::1]:123 2024-06-27T21:29:33.770 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1440%4]:123 2024-06-27T21:29:33.770 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:33 ntpd[17475]: Listening on routing socket on fd #22 for interface updates 2024-06-27T21:29:34.767 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:34.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:34.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:34.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:34.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:34.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:34.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-27T21:29:34.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: can't open /var/log/ntpstats/peerstats.20240627: Permission denied 2024-06-27T21:29:34.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:34.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:34.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-27T21:29:34.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:34 ntpd[17475]: can't open /var/log/ntpstats/peerstats.20240627: Permission denied 2024-06-27T21:29:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-27T21:29:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: can't open /var/log/ntpstats/peerstats.20240627: Permission denied 2024-06-27T21:29:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-27T21:29:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:36 ntpd[17475]: can't open /var/log/ntpstats/peerstats.20240627: Permission denied 2024-06-27T21:29:37.767 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:37 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:37.767 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:37 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:38.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:38.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:38.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:38.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: can't open /var/log/ntpstats/peerstats.20240627: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: can't open /var/log/ntpstats/peerstats.20240627: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-27T21:29:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:38 ntpd[17475]: can't open /var/log/ntpstats/peerstats.20240627: Permission denied 2024-06-27T21:29:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-27T21:29:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: can't open /var/log/ntpstats/rawstats.20240627: Permission denied 2024-06-27T21:29:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-27T21:29:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: can't open /var/log/ntpstats/peerstats.20240627: Permission denied 2024-06-27T21:29:40.769 INFO:teuthology.orchestra.run.smithi042.stdout:27 Jun 21:29:40 ntpd[17475]: ntpd: time slew +0.001667 s 2024-06-27T21:29:40.769 INFO:teuthology.orchestra.run.smithi042.stdout:ntpd: time slew +0.001667s 2024-06-27T21:29:40.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-06-27T21:29:40.769 INFO:teuthology.orchestra.run.smithi042.stderr:27 Jun 21:29:40 ntpd[17475]: can't open /var/log/ntpstats/loopstats.20240627: Permission denied 2024-06-27T21:29:40.822 INFO:teuthology.orchestra.run.smithi042.stdout: remote refid st t when poll reach delay offset jitter 2024-06-27T21:29:40.822 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================== 2024-06-27T21:29:40.822 INFO:teuthology.orchestra.run.smithi042.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-27T21:29:40.822 INFO:teuthology.orchestra.run.smithi042.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-27T21:29:40.822 INFO:teuthology.orchestra.run.smithi042.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-27T21:29:40.823 INFO:teuthology.orchestra.run.smithi042.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-27T21:29:40.824 INFO:teuthology.run_tasks:Running task install... 2024-06-27T21:29:40.833 DEBUG:teuthology.task.install:project ceph 2024-06-27T21:29:40.833 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122'}} 2024-06-27T21:29:40.833 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122'} 2024-06-27T21:29:40.833 INFO:teuthology.task.install:Using flavor: default 2024-06-27T21:29:40.838 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-27T21:29:40.838 INFO:teuthology.task.install:extra packages: [] 2024-06-27T21:29:40.839 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-key list | grep Ceph 2024-06-27T21:29:40.925 INFO:teuthology.orchestra.run.smithi042.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-06-27T21:29:40.955 INFO:teuthology.orchestra.run.smithi042.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-06-27T21:29:40.955 INFO:teuthology.orchestra.run.smithi042.stdout:uid [ unknown] Ceph.com (release key) 2024-06-27T21:29:40.956 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-06-27T21:29:40.956 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:29:41.184 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default/ 2024-06-27T21:29:41.185 INFO:teuthology.task.install.deb:Package version is 19.0.0-4578-gf51acc18-1jammy 2024-06-27T21:29:41.292 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-06-27T21:29:41.292 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-06-27T21:29:41.307 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get update 2024-06-27T21:29:41.483 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-27T21:29:41.483 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-27T21:29:41.495 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-06-27T21:29:41.516 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-27T21:29:41.591 INFO:teuthology.orchestra.run.smithi042.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy InRelease 2024-06-27T21:29:41.627 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-06-27T21:29:41.664 INFO:teuthology.orchestra.run.smithi042.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy Release.gpg 2024-06-27T21:29:41.700 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-06-27T21:29:41.742 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-06-27T21:29:41.849 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 29.2 kB in 0s (81.1 kB/s) 2024-06-27T21:29:42.867 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-06-27T21:29:42.885 DEBUG:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-4578-gf51acc18-1jammy cephadm=19.0.0-4578-gf51acc18-1jammy ceph-mds=19.0.0-4578-gf51acc18-1jammy ceph-mgr=19.0.0-4578-gf51acc18-1jammy ceph-common=19.0.0-4578-gf51acc18-1jammy ceph-fuse=19.0.0-4578-gf51acc18-1jammy ceph-test=19.0.0-4578-gf51acc18-1jammy ceph-volume=19.0.0-4578-gf51acc18-1jammy radosgw=19.0.0-4578-gf51acc18-1jammy python3-rados=19.0.0-4578-gf51acc18-1jammy python3-rgw=19.0.0-4578-gf51acc18-1jammy python3-cephfs=19.0.0-4578-gf51acc18-1jammy python3-rbd=19.0.0-4578-gf51acc18-1jammy libcephfs2=19.0.0-4578-gf51acc18-1jammy libcephfs-dev=19.0.0-4578-gf51acc18-1jammy librados2=19.0.0-4578-gf51acc18-1jammy librbd1=19.0.0-4578-gf51acc18-1jammy rbd-fuse=19.0.0-4578-gf51acc18-1jammy 2024-06-27T21:29:42.951 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-06-27T21:29:43.164 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2024-06-27T21:29:43.165 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2024-06-27T21:29:43.377 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2024-06-27T21:29:43.377 INFO:teuthology.orchestra.run.smithi042.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-27T21:29:43.378 INFO:teuthology.orchestra.run.smithi042.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-27T21:29:43.378 INFO:teuthology.orchestra.run.smithi042.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-27T21:29:43.378 INFO:teuthology.orchestra.run.smithi042.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-27T21:29:43.378 INFO:teuthology.orchestra.run.smithi042.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-27T21:29:43.378 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-27T21:29:43.379 INFO:teuthology.orchestra.run.smithi042.stdout:The following additional packages will be installed: 2024-06-27T21:29:43.379 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-06-27T21:29:43.379 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-06-27T21:29:43.379 INFO:teuthology.orchestra.run.smithi042.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-06-27T21:29:43.379 INFO:teuthology.orchestra.run.smithi042.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-06-27T21:29:43.382 INFO:teuthology.orchestra.run.smithi042.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-06-27T21:29:43.382 INFO:teuthology.orchestra.run.smithi042.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-06-27T21:29:43.382 INFO:teuthology.orchestra.run.smithi042.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-06-27T21:29:43.382 INFO:teuthology.orchestra.run.smithi042.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-06-27T21:29:43.382 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-06-27T21:29:43.382 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-06-27T21:29:43.382 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-06-27T21:29:43.382 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-06-27T21:29:43.382 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-06-27T21:29:43.382 INFO:teuthology.orchestra.run.smithi042.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-06-27T21:29:43.383 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-06-27T21:29:43.383 INFO:teuthology.orchestra.run.smithi042.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-06-27T21:29:43.383 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-06-27T21:29:43.383 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-06-27T21:29:43.383 INFO:teuthology.orchestra.run.smithi042.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-06-27T21:29:43.383 INFO:teuthology.orchestra.run.smithi042.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-06-27T21:29:43.383 INFO:teuthology.orchestra.run.smithi042.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-06-27T21:29:43.383 INFO:teuthology.orchestra.run.smithi042.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-06-27T21:29:43.383 INFO:teuthology.orchestra.run.smithi042.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-06-27T21:29:43.383 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet zip 2024-06-27T21:29:43.383 INFO:teuthology.orchestra.run.smithi042.stdout:Suggested packages: 2024-06-27T21:29:43.384 INFO:teuthology.orchestra.run.smithi042.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-06-27T21:29:43.384 INFO:teuthology.orchestra.run.smithi042.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-06-27T21:29:43.384 INFO:teuthology.orchestra.run.smithi042.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-06-27T21:29:43.384 INFO:teuthology.orchestra.run.smithi042.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-06-27T21:29:43.384 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-06-27T21:29:43.384 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-06-27T21:29:43.384 INFO:teuthology.orchestra.run.smithi042.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-06-27T21:29:43.384 INFO:teuthology.orchestra.run.smithi042.stdout:Recommended packages: 2024-06-27T21:29:43.384 INFO:teuthology.orchestra.run.smithi042.stdout: btrfs-tools 2024-06-27T21:29:43.491 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2024-06-27T21:29:43.491 INFO:teuthology.orchestra.run.smithi042.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-06-27T21:29:43.491 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-06-27T21:29:43.491 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-06-27T21:29:43.491 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-06-27T21:29:43.491 INFO:teuthology.orchestra.run.smithi042.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-06-27T21:29:43.492 INFO:teuthology.orchestra.run.smithi042.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-06-27T21:29:43.493 INFO:teuthology.orchestra.run.smithi042.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-06-27T21:29:43.493 INFO:teuthology.orchestra.run.smithi042.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-06-27T21:29:43.493 INFO:teuthology.orchestra.run.smithi042.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-06-27T21:29:43.493 INFO:teuthology.orchestra.run.smithi042.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-06-27T21:29:43.493 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-06-27T21:29:43.493 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-06-27T21:29:43.493 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-06-27T21:29:43.493 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-06-27T21:29:43.493 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-06-27T21:29:43.493 INFO:teuthology.orchestra.run.smithi042.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-06-27T21:29:43.494 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-06-27T21:29:43.494 INFO:teuthology.orchestra.run.smithi042.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-06-27T21:29:43.494 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-06-27T21:29:43.494 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-06-27T21:29:43.494 INFO:teuthology.orchestra.run.smithi042.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-06-27T21:29:43.494 INFO:teuthology.orchestra.run.smithi042.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-06-27T21:29:43.494 INFO:teuthology.orchestra.run.smithi042.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-06-27T21:29:43.494 INFO:teuthology.orchestra.run.smithi042.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-06-27T21:29:43.494 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-06-27T21:29:43.494 INFO:teuthology.orchestra.run.smithi042.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-06-27T21:29:43.636 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 118 newly installed, 0 to remove and 315 not upgraded. 2024-06-27T21:29:43.637 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 248 MB of archives. 2024-06-27T21:29:43.637 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 916 MB of additional disk space will be used. 2024-06-27T21:29:43.637 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-4578-gf51acc18-1jammy [3,571 kB] 2024-06-27T21:29:43.682 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-06-27T21:29:43.830 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-4578-gf51acc18-1jammy [3,231 kB] 2024-06-27T21:29:43.889 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-4578-gf51acc18-1jammy [716 kB] 2024-06-27T21:29:43.901 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-4578-gf51acc18-1jammy [356 kB] 2024-06-27T21:29:43.907 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-4578-gf51acc18-1jammy [32.8 kB] 2024-06-27T21:29:43.908 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-4578-gf51acc18-1jammy [180 kB] 2024-06-27T21:29:43.911 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-4578-gf51acc18-1jammy [67.4 kB] 2024-06-27T21:29:43.912 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-4578-gf51acc18-1jammy [334 kB] 2024-06-27T21:29:43.917 INFO:teuthology.orchestra.run.smithi042.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-4578-gf51acc18-1jammy [7,891 kB] 2024-06-27T21:29:44.040 INFO:teuthology.orchestra.run.smithi042.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-4578-gf51acc18-1jammy [112 kB] 2024-06-27T21:29:44.042 INFO:teuthology.orchestra.run.smithi042.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-4578-gf51acc18-1jammy [470 kB] 2024-06-27T21:29:44.048 INFO:teuthology.orchestra.run.smithi042.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-4578-gf51acc18-1jammy [27.9 MB] 2024-06-27T21:29:44.106 INFO:teuthology.orchestra.run.smithi042.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-06-27T21:29:44.118 INFO:teuthology.orchestra.run.smithi042.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-06-27T21:29:44.199 INFO:teuthology.orchestra.run.smithi042.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-06-27T21:29:44.428 INFO:teuthology.orchestra.run.smithi042.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-06-27T21:29:44.439 INFO:teuthology.orchestra.run.smithi042.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-06-27T21:29:44.469 INFO:teuthology.orchestra.run.smithi042.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-06-27T21:29:44.477 INFO:teuthology.orchestra.run.smithi042.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-06-27T21:29:44.478 INFO:teuthology.orchestra.run.smithi042.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-06-27T21:29:44.480 INFO:teuthology.orchestra.run.smithi042.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-06-27T21:29:44.499 INFO:teuthology.orchestra.run.smithi042.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-06-27T21:29:44.503 INFO:teuthology.orchestra.run.smithi042.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-06-27T21:29:44.507 INFO:teuthology.orchestra.run.smithi042.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-06-27T21:29:44.570 INFO:teuthology.orchestra.run.smithi042.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-06-27T21:29:44.592 INFO:teuthology.orchestra.run.smithi042.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-06-27T21:29:44.628 INFO:teuthology.orchestra.run.smithi042.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-06-27T21:29:44.629 INFO:teuthology.orchestra.run.smithi042.stdout:Get:29 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-4578-gf51acc18-1jammy [5,096 kB] 2024-06-27T21:29:44.633 INFO:teuthology.orchestra.run.smithi042.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-06-27T21:29:44.636 INFO:teuthology.orchestra.run.smithi042.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-06-27T21:29:44.638 INFO:teuthology.orchestra.run.smithi042.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-06-27T21:29:44.638 INFO:teuthology.orchestra.run.smithi042.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-06-27T21:29:44.640 INFO:teuthology.orchestra.run.smithi042.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-06-27T21:29:44.674 INFO:teuthology.orchestra.run.smithi042.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-06-27T21:29:44.674 INFO:teuthology.orchestra.run.smithi042.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-06-27T21:29:44.675 INFO:teuthology.orchestra.run.smithi042.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-06-27T21:29:44.695 INFO:teuthology.orchestra.run.smithi042.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-4578-gf51acc18-1jammy [247 kB] 2024-06-27T21:29:44.697 INFO:teuthology.orchestra.run.smithi042.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-4578-gf51acc18-1jammy [124 kB] 2024-06-27T21:29:44.699 INFO:teuthology.orchestra.run.smithi042.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-4578-gf51acc18-1jammy [1,547 kB] 2024-06-27T21:29:44.713 INFO:teuthology.orchestra.run.smithi042.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-4578-gf51acc18-1jammy [6,268 kB] 2024-06-27T21:29:44.755 INFO:teuthology.orchestra.run.smithi042.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-06-27T21:29:44.756 INFO:teuthology.orchestra.run.smithi042.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-06-27T21:29:44.756 INFO:teuthology.orchestra.run.smithi042.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-06-27T21:29:44.761 INFO:teuthology.orchestra.run.smithi042.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-06-27T21:29:44.762 INFO:teuthology.orchestra.run.smithi042.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-06-27T21:29:44.764 INFO:teuthology.orchestra.run.smithi042.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-06-27T21:29:44.764 INFO:teuthology.orchestra.run.smithi042.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-06-27T21:29:44.801 INFO:teuthology.orchestra.run.smithi042.stdout:Get:49 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-4578-gf51acc18-1jammy [22.8 MB] 2024-06-27T21:29:44.836 INFO:teuthology.orchestra.run.smithi042.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-06-27T21:29:44.836 INFO:teuthology.orchestra.run.smithi042.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-06-27T21:29:44.837 INFO:teuthology.orchestra.run.smithi042.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-06-27T21:29:44.917 INFO:teuthology.orchestra.run.smithi042.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-06-27T21:29:44.919 INFO:teuthology.orchestra.run.smithi042.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-06-27T21:29:44.919 INFO:teuthology.orchestra.run.smithi042.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-06-27T21:29:44.931 INFO:teuthology.orchestra.run.smithi042.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-06-27T21:29:44.931 INFO:teuthology.orchestra.run.smithi042.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-06-27T21:29:44.932 INFO:teuthology.orchestra.run.smithi042.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-06-27T21:29:44.933 INFO:teuthology.orchestra.run.smithi042.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-06-27T21:29:44.999 INFO:teuthology.orchestra.run.smithi042.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-06-27T21:29:45.003 INFO:teuthology.orchestra.run.smithi042.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-06-27T21:29:45.004 INFO:teuthology.orchestra.run.smithi042.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-06-27T21:29:45.080 INFO:teuthology.orchestra.run.smithi042.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-06-27T21:29:45.082 INFO:teuthology.orchestra.run.smithi042.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-06-27T21:29:45.087 INFO:teuthology.orchestra.run.smithi042.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-06-27T21:29:45.088 INFO:teuthology.orchestra.run.smithi042.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-06-27T21:29:45.090 INFO:teuthology.orchestra.run.smithi042.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-06-27T21:29:45.095 INFO:teuthology.orchestra.run.smithi042.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-06-27T21:29:45.251 INFO:teuthology.orchestra.run.smithi042.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-4578-gf51acc18-1jammy [14.1 kB] 2024-06-27T21:29:45.252 INFO:teuthology.orchestra.run.smithi042.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-4578-gf51acc18-1jammy [871 kB] 2024-06-27T21:29:45.317 INFO:teuthology.orchestra.run.smithi042.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-4578-gf51acc18-1jammy [2,446 kB] 2024-06-27T21:29:45.317 INFO:teuthology.orchestra.run.smithi042.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-06-27T21:29:45.319 INFO:teuthology.orchestra.run.smithi042.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-06-27T21:29:45.322 INFO:teuthology.orchestra.run.smithi042.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-06-27T21:29:45.323 INFO:teuthology.orchestra.run.smithi042.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-06-27T21:29:45.326 INFO:teuthology.orchestra.run.smithi042.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-06-27T21:29:45.349 INFO:teuthology.orchestra.run.smithi042.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-4578-gf51acc18-1jammy [793 kB] 2024-06-27T21:29:45.359 INFO:teuthology.orchestra.run.smithi042.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-4578-gf51acc18-1jammy [149 kB] 2024-06-27T21:29:45.361 INFO:teuthology.orchestra.run.smithi042.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-4578-gf51acc18-1jammy [60.3 MB] 2024-06-27T21:29:45.371 INFO:teuthology.orchestra.run.smithi042.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-06-27T21:29:45.374 INFO:teuthology.orchestra.run.smithi042.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-06-27T21:29:45.375 INFO:teuthology.orchestra.run.smithi042.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-06-27T21:29:45.419 INFO:teuthology.orchestra.run.smithi042.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-06-27T21:29:45.420 INFO:teuthology.orchestra.run.smithi042.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-06-27T21:29:45.484 INFO:teuthology.orchestra.run.smithi042.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-06-27T21:29:45.485 INFO:teuthology.orchestra.run.smithi042.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-06-27T21:29:45.486 INFO:teuthology.orchestra.run.smithi042.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-06-27T21:29:45.488 INFO:teuthology.orchestra.run.smithi042.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-06-27T21:29:45.489 INFO:teuthology.orchestra.run.smithi042.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-06-27T21:29:45.491 INFO:teuthology.orchestra.run.smithi042.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-06-27T21:29:45.499 INFO:teuthology.orchestra.run.smithi042.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-06-27T21:29:45.565 INFO:teuthology.orchestra.run.smithi042.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-06-27T21:29:45.569 INFO:teuthology.orchestra.run.smithi042.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-06-27T21:29:45.570 INFO:teuthology.orchestra.run.smithi042.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-06-27T21:29:45.647 INFO:teuthology.orchestra.run.smithi042.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-06-27T21:29:45.653 INFO:teuthology.orchestra.run.smithi042.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-06-27T21:29:45.655 INFO:teuthology.orchestra.run.smithi042.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-06-27T21:29:45.657 INFO:teuthology.orchestra.run.smithi042.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-06-27T21:29:45.668 INFO:teuthology.orchestra.run.smithi042.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-06-27T21:29:45.669 INFO:teuthology.orchestra.run.smithi042.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-06-27T21:29:45.677 INFO:teuthology.orchestra.run.smithi042.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-06-27T21:29:45.728 INFO:teuthology.orchestra.run.smithi042.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-06-27T21:29:45.730 INFO:teuthology.orchestra.run.smithi042.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-06-27T21:29:45.731 INFO:teuthology.orchestra.run.smithi042.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-06-27T21:29:45.810 INFO:teuthology.orchestra.run.smithi042.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-06-27T21:29:45.812 INFO:teuthology.orchestra.run.smithi042.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-06-27T21:29:45.829 INFO:teuthology.orchestra.run.smithi042.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-06-27T21:29:45.830 INFO:teuthology.orchestra.run.smithi042.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-06-27T21:29:45.831 INFO:teuthology.orchestra.run.smithi042.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-06-27T21:29:45.836 INFO:teuthology.orchestra.run.smithi042.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-06-27T21:29:45.838 INFO:teuthology.orchestra.run.smithi042.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-06-27T21:29:46.834 INFO:teuthology.orchestra.run.smithi042.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-4578-gf51acc18-1jammy [8,625 kB] 2024-06-27T21:29:46.983 INFO:teuthology.orchestra.run.smithi042.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-4578-gf51acc18-1jammy [14.2 kB] 2024-06-27T21:29:46.983 INFO:teuthology.orchestra.run.smithi042.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-4578-gf51acc18-1jammy [54.8 MB] 2024-06-27T21:29:48.528 INFO:teuthology.orchestra.run.smithi042.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-4578-gf51acc18-1jammy [125 kB] 2024-06-27T21:29:48.531 INFO:teuthology.orchestra.run.smithi042.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-4578-gf51acc18-1jammy [38.9 kB] 2024-06-27T21:29:48.532 INFO:teuthology.orchestra.run.smithi042.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-4578-gf51acc18-1jammy [15.7 MB] 2024-06-27T21:29:48.889 INFO:teuthology.orchestra.run.smithi042.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-4578-gf51acc18-1jammy [91.3 kB] 2024-06-27T21:29:49.487 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 248 MB in 5s (45.9 MB/s) 2024-06-27T21:29:49.624 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-06-27T21:29:49.726 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165982 files and directories currently installed.) 2024-06-27T21:29:49.732 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-06-27T21:29:49.805 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-27T21:29:50.134 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-06-27T21:29:50.162 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-06-27T21:29:50.201 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-27T21:29:50.546 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-06-27T21:29:50.573 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-06-27T21:29:50.612 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-27T21:29:51.058 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-06-27T21:29:51.085 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-27T21:29:51.133 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-27T21:29:51.503 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-06-27T21:29:51.530 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-27T21:29:51.569 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-27T21:29:51.931 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-06-27T21:29:51.958 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-27T21:29:52.006 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-27T21:29:52.509 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-06-27T21:29:52.536 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-06-27T21:29:52.575 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-27T21:29:52.871 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librados2. 2024-06-27T21:29:52.887 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../007-librados2_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:29:52.920 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librados2 (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:29:53.365 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librbd1. 2024-06-27T21:29:53.392 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../008-librbd1_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:29:53.448 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librbd1 (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:29:53.893 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libcephfs2. 2024-06-27T21:29:53.920 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:29:53.959 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libcephfs2 (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:29:54.246 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rados. 2024-06-27T21:29:54.262 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../010-python3-rados_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:29:54.296 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rados (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:29:54.590 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-06-27T21:29:54.617 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-4578-gf51acc18-1jammy_all.deb ... 2024-06-27T21:29:54.657 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-ceph-argparse (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:29:54.909 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cephfs. 2024-06-27T21:29:54.926 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:29:54.967 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cephfs (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:29:55.287 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-ceph-common. 2024-06-27T21:29:55.314 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-4578-gf51acc18-1jammy_all.deb ... 2024-06-27T21:29:55.354 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-ceph-common (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:29:55.665 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-wcwidth. 2024-06-27T21:29:55.682 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-06-27T21:29:55.715 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-27T21:29:56.044 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-prettytable. 2024-06-27T21:29:56.071 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-06-27T21:29:56.110 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-06-27T21:29:56.388 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rbd. 2024-06-27T21:29:56.404 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:29:56.446 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rbd (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:29:56.774 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-06-27T21:29:56.801 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-06-27T21:29:57.058 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-27T21:29:57.302 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-06-27T21:29:57.318 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-06-27T21:29:57.352 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-06-27T21:29:57.747 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-06-27T21:29:57.774 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-06-27T21:29:57.813 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-06-27T21:29:58.142 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua5.1. 2024-06-27T21:29:58.169 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-06-27T21:29:58.208 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-06-27T21:29:58.520 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua-any. 2024-06-27T21:29:58.547 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-06-27T21:29:58.578 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua-any (27ubuntu1) ... 2024-06-27T21:29:58.814 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package zip. 2024-06-27T21:29:58.841 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-06-27T21:29:58.881 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking zip (3.0-12build2) ... 2024-06-27T21:29:59.276 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package unzip. 2024-06-27T21:29:59.303 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-06-27T21:29:59.351 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-06-27T21:29:59.780 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package luarocks. 2024-06-27T21:29:59.807 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-06-27T21:29:59.847 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-06-27T21:30:00.264 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librgw2. 2024-06-27T21:30:00.291 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../025-librgw2_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:00.338 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librgw2 (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:00.999 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rgw. 2024-06-27T21:30:01.052 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:01.074 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rgw (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:01.427 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package liboath0:amd64. 2024-06-27T21:30:01.455 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-06-27T21:30:01.494 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-06-27T21:30:01.814 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libradosstriper1. 2024-06-27T21:30:01.841 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:01.880 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libradosstriper1 (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:02.434 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-common. 2024-06-27T21:30:02.462 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../029-ceph-common_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:02.509 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-common (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:04.090 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-base. 2024-06-27T21:30:04.117 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../030-ceph-base_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:04.167 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-base (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:04.728 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-06-27T21:30:04.755 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-06-27T21:30:04.803 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-06-27T21:30:05.182 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cheroot. 2024-06-27T21:30:05.208 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-06-27T21:30:05.278 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-27T21:30:05.617 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-06-27T21:30:05.644 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-06-27T21:30:05.684 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-06-27T21:30:05.987 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.text. 2024-06-27T21:30:06.014 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-06-27T21:30:06.053 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-06-27T21:30:06.390 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-06-27T21:30:06.417 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-06-27T21:30:06.465 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-06-27T21:30:06.835 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-tempora. 2024-06-27T21:30:06.862 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-06-27T21:30:06.902 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-06-27T21:30:07.446 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-portend. 2024-06-27T21:30:07.474 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-06-27T21:30:07.513 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-06-27T21:30:07.874 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-06-27T21:30:07.902 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-06-27T21:30:07.950 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-06-27T21:30:08.302 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cherrypy3. 2024-06-27T21:30:08.330 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-06-27T21:30:08.377 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-06-27T21:30:08.723 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-natsort. 2024-06-27T21:30:08.750 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-06-27T21:30:08.789 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-06-27T21:30:09.076 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-logutils. 2024-06-27T21:30:09.092 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-06-27T21:30:09.134 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-06-27T21:30:09.437 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-markupsafe. 2024-06-27T21:30:09.464 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-06-27T21:30:09.504 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-06-27T21:30:09.807 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-mako. 2024-06-27T21:30:09.834 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-06-27T21:30:09.873 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-27T21:30:10.178 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-simplegeneric. 2024-06-27T21:30:10.204 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-06-27T21:30:10.243 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-06-27T21:30:10.513 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-singledispatch. 2024-06-27T21:30:10.541 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-06-27T21:30:10.580 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-06-27T21:30:10.899 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-webob. 2024-06-27T21:30:10.927 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-06-27T21:30:10.966 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-06-27T21:30:11.302 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-waitress. 2024-06-27T21:30:11.330 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-06-27T21:30:11.372 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-27T21:30:11.714 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-tempita. 2024-06-27T21:30:11.741 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-06-27T21:30:11.780 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-06-27T21:30:12.042 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-paste. 2024-06-27T21:30:12.069 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-06-27T21:30:12.100 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-06-27T21:30:12.420 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-06-27T21:30:12.436 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-06-27T21:30:12.478 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-06-27T21:30:12.748 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pastedeploy. 2024-06-27T21:30:12.776 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-06-27T21:30:12.806 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-06-27T21:30:13.109 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-webtest. 2024-06-27T21:30:13.137 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-06-27T21:30:13.176 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-06-27T21:30:13.488 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pecan. 2024-06-27T21:30:13.515 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-06-27T21:30:13.554 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-06-27T21:30:13.919 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-certifi. 2024-06-27T21:30:13.947 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-06-27T21:30:13.986 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-06-27T21:30:14.280 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-idna. 2024-06-27T21:30:14.299 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-06-27T21:30:14.330 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-06-27T21:30:14.650 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-urllib3. 2024-06-27T21:30:14.666 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-06-27T21:30:14.709 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-27T21:30:15.028 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-requests. 2024-06-27T21:30:15.055 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-06-27T21:30:15.094 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-27T21:30:15.422 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-werkzeug. 2024-06-27T21:30:15.450 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-06-27T21:30:15.489 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-27T21:30:15.784 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-06-27T21:30:15.800 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-4578-gf51acc18-1jammy_all.deb ... 2024-06-27T21:30:15.842 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-modules-core (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:16.239 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-06-27T21:30:16.266 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:16.305 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:16.600 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-bcrypt. 2024-06-27T21:30:16.627 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-06-27T21:30:16.666 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-06-27T21:30:17.803 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-openssl. 2024-06-27T21:30:17.831 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-06-27T21:30:17.878 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-06-27T21:30:18.239 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr. 2024-06-27T21:30:18.267 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:18.305 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:18.718 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mon. 2024-06-27T21:30:18.745 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:18.784 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mon (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:19.438 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-osd. 2024-06-27T21:30:19.465 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:19.504 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-osd (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:20.641 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph. 2024-06-27T21:30:20.657 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../066-ceph_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:20.716 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:21.060 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-fuse. 2024-06-27T21:30:21.088 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:21.126 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-fuse (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:21.522 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mds. 2024-06-27T21:30:21.549 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:21.588 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mds (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:22.049 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package cephadm. 2024-06-27T21:30:22.077 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../069-cephadm_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:22.133 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking cephadm (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:22.477 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-asyncssh. 2024-06-27T21:30:22.505 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-06-27T21:30:22.543 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-06-27T21:30:22.964 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-babel-localedata. 2024-06-27T21:30:22.991 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-06-27T21:30:23.038 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-27T21:30:24.010 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-babel. 2024-06-27T21:30:24.038 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-06-27T21:30:24.068 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-06-27T21:30:24.379 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jinja2. 2024-06-27T21:30:24.407 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-06-27T21:30:24.446 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-27T21:30:24.782 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-06-27T21:30:24.799 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-4578-gf51acc18-1jammy_all.deb ... 2024-06-27T21:30:24.840 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-cephadm (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:25.169 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-repoze.lru. 2024-06-27T21:30:25.186 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-06-27T21:30:25.227 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-06-27T21:30:25.513 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-routes. 2024-06-27T21:30:25.541 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-06-27T21:30:25.580 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-06-27T21:30:25.934 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-06-27T21:30:25.961 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-4578-gf51acc18-1jammy_all.deb ... 2024-06-27T21:30:26.000 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-dashboard (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:28.996 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-06-27T21:30:29.015 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-06-27T21:30:29.054 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-27T21:30:29.508 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-joblib. 2024-06-27T21:30:29.524 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-06-27T21:30:29.566 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-06-27T21:30:29.936 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-06-27T21:30:29.965 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-06-27T21:30:30.003 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-06-27T21:30:30.306 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-sklearn. 2024-06-27T21:30:30.334 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-06-27T21:30:30.372 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-27T21:30:31.047 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-06-27T21:30:31.065 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-4578-gf51acc18-1jammy_all.deb ... 2024-06-27T21:30:31.191 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:32.145 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cachetools. 2024-06-27T21:30:32.173 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-06-27T21:30:32.211 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-06-27T21:30:32.456 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pyasn1. 2024-06-27T21:30:32.473 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-06-27T21:30:32.506 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-06-27T21:30:32.818 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-06-27T21:30:32.846 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-06-27T21:30:32.884 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rsa. 2024-06-27T21:30:33.233 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rsa (4.8-1) ... 2024-06-27T21:30:33.599 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-google-auth. 2024-06-27T21:30:33.627 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-06-27T21:30:33.665 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-06-27T21:30:33.969 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-06-27T21:30:33.997 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-06-27T21:30:34.035 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-27T21:30:34.347 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-websocket. 2024-06-27T21:30:34.375 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-06-27T21:30:34.413 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-06-27T21:30:34.758 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-kubernetes. 2024-06-27T21:30:34.786 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-06-27T21:30:34.825 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-27T21:30:35.522 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-06-27T21:30:35.539 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-4578-gf51acc18-1jammy_all.deb ... 2024-06-27T21:30:35.580 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:35.900 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libonig5:amd64. 2024-06-27T21:30:35.928 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-06-27T21:30:35.966 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-27T21:30:36.269 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libjq1:amd64. 2024-06-27T21:30:36.297 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-06-27T21:30:36.335 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-27T21:30:36.588 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package jq. 2024-06-27T21:30:36.617 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-06-27T21:30:36.655 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-06-27T21:30:36.891 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package socat. 2024-06-27T21:30:36.919 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-06-27T21:30:36.957 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-06-27T21:30:37.269 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package xmlstarlet. 2024-06-27T21:30:37.286 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-06-27T21:30:37.327 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-06-27T21:30:37.690 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-test. 2024-06-27T21:30:37.718 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../097-ceph-test_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:37.756 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-test (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:40.228 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-volume. 2024-06-27T21:30:40.256 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-4578-gf51acc18-1jammy_all.deb ... 2024-06-27T21:30:40.294 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-volume (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:40.606 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libcephfs-dev. 2024-06-27T21:30:40.635 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:40.672 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libcephfs-dev (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:40.967 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua-socket:amd64. 2024-06-27T21:30:40.995 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-06-27T21:30:41.034 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-06-27T21:30:41.355 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua-sec:amd64. 2024-06-27T21:30:41.384 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-06-27T21:30:41.421 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-06-27T21:30:41.817 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package nvme-cli. 2024-06-27T21:30:41.846 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-06-27T21:30:41.945 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-06-27T21:30:42.314 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package pkg-config. 2024-06-27T21:30:42.343 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-06-27T21:30:42.380 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-06-27T21:30:42.617 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-06-27T21:30:42.633 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-06-27T21:30:42.675 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-06-27T21:30:43.112 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-iniconfig. 2024-06-27T21:30:43.140 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-06-27T21:30:43.170 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-06-27T21:30:43.714 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pastescript. 2024-06-27T21:30:43.743 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-06-27T21:30:43.780 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-06-27T21:30:44.084 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pluggy. 2024-06-27T21:30:44.113 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-06-27T21:30:44.150 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-06-27T21:30:44.445 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-psutil. 2024-06-27T21:30:44.474 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-06-27T21:30:44.511 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-06-27T21:30:44.857 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-py. 2024-06-27T21:30:44.885 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-06-27T21:30:44.923 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-py (1.10.0-1) ... 2024-06-27T21:30:45.209 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pygments. 2024-06-27T21:30:45.226 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-06-27T21:30:45.268 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-06-27T21:30:45.705 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pyinotify. 2024-06-27T21:30:45.734 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-06-27T21:30:45.771 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-06-27T21:30:46.116 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-toml. 2024-06-27T21:30:46.145 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-06-27T21:30:46.182 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-06-27T21:30:46.486 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pytest. 2024-06-27T21:30:46.514 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-06-27T21:30:46.552 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-06-27T21:30:46.897 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-simplejson. 2024-06-27T21:30:46.926 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-06-27T21:30:46.963 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-06-27T21:30:47.233 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-06-27T21:30:47.250 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-06-27T21:30:47.291 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-06-27T21:30:47.987 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package radosgw. 2024-06-27T21:30:48.016 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../116-radosgw_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:48.045 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking radosgw (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:48.998 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package rbd-fuse. 2024-06-27T21:30:49.027 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-4578-gf51acc18-1jammy_amd64.deb ... 2024-06-27T21:30:49.065 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking rbd-fuse (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:49.378 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-06-27T21:30:49.636 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-27T21:30:49.762 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-06-27T21:30:50.066 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-06-27T21:30:50.382 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-06-27T21:30:50.807 INFO:teuthology.orchestra.run.smithi042.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-06-27T21:30:50.881 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up cephadm (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:51.406 INFO:teuthology.orchestra.run.smithi042.stdout:Adding system user cephadm....done 2024-06-27T21:30:51.562 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-27T21:30:51.847 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-06-27T21:30:52.089 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-06-27T21:30:52.190 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-06-27T21:30:52.425 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-06-27T21:30:52.684 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-06-27T21:30:52.819 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-py (1.10.0-1) ... 2024-06-27T21:30:53.121 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-06-27T21:30:53.506 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-06-27T21:30:53.757 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-06-27T21:30:54.025 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-06-27T21:30:54.277 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-06-27T21:30:54.536 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-06-27T21:30:54.795 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-06-27T21:30:55.072 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-ceph-argparse (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:30:55.347 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-27T21:30:55.465 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-06-27T21:30:55.566 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-06-27T21:30:55.675 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-06-27T21:30:55.935 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-06-27T21:30:56.324 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-06-27T21:30:56.349 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-06-27T21:30:56.405 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-27T21:30:56.531 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-06-27T21:30:56.882 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-27T21:30:56.983 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-06-27T21:30:57.242 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-06-27T21:30:57.537 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-06-27T21:30:57.847 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up zip (3.0-12build2) ... 2024-06-27T21:30:57.973 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-06-27T21:30:58.633 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-06-27T21:30:58.884 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-06-27T21:30:59.010 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-06-27T21:30:59.411 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-27T21:30:59.720 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-06-27T21:31:00.122 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-06-27T21:31:00.373 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-06-27T21:31:00.741 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-27T21:31:01.025 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-27T21:31:01.377 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-06-27T21:31:01.628 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-06-27T21:31:01.762 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-ceph-common (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:02.072 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-06-27T21:31:02.381 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-27T21:31:03.579 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-06-27T21:31:03.839 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-27T21:31:03.948 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-toml (0.10.2-1) ... 2024-06-27T21:31:04.199 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-27T21:31:04.300 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-06-27T21:31:04.426 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-27T21:31:04.727 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-06-27T21:31:05.004 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-06-27T21:31:05.263 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-27T21:31:05.389 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-06-27T21:31:05.715 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-06-27T21:31:05.958 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-06-27T21:31:06.234 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-06-27T21:31:06.494 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-06-27T21:31:06.753 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-06-27T21:31:07.021 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-27T21:31:07.147 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-06-27T21:31:07.439 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-27T21:31:07.540 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-27T21:31:07.817 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-06-27T21:31:08.118 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-06-27T21:31:08.385 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-06-27T21:31:08.511 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-06-27T21:31:08.646 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-06-27T21:31:08.736 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-06-27T21:31:09.015 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-27T21:31:09.123 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-06-27T21:31:09.491 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-06-27T21:31:09.759 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua-any (27ubuntu1) ... 2024-06-27T21:31:09.885 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-portend (3.0.0-1) ... 2024-06-27T21:31:10.153 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-27T21:31:10.287 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-27T21:31:10.622 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-06-27T21:31:10.915 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-27T21:31:11.174 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-06-27T21:31:11.300 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rsa (4.8-1) ... 2024-06-27T21:31:11.585 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-06-27T21:31:11.852 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-27T21:31:12.206 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-06-27T21:31:12.574 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-06-27T21:31:12.892 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-06-27T21:31:13.228 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-27T21:31:13.362 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librados2 (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:13.472 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libsqlite3-mod-ceph (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:13.573 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-06-27T21:31:13.901 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libcephfs2 (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:14.027 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libradosstriper1 (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:14.153 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-06-27T21:31:14.480 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librbd1 (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:14.597 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-modules-core (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:14.707 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-fuse (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:14.910 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-27T21:31:14.910 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-27T21:31:15.425 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libcephfs-dev (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:15.551 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rados (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:15.677 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librgw2 (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:15.803 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-27T21:31:17.090 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rbd (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:17.216 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up rbd-fuse (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:17.342 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rgw (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:17.468 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cephfs (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:17.594 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-common (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:18.103 INFO:teuthology.orchestra.run.smithi042.stdout:Adding group ceph....done 2024-06-27T21:31:18.439 INFO:teuthology.orchestra.run.smithi042.stdout:Adding system user ceph....done 2024-06-27T21:31:18.607 INFO:teuthology.orchestra.run.smithi042.stdout:Setting system user ceph properties....done 2024-06-27T21:31:18.620 INFO:teuthology.orchestra.run.smithi042.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-06-27T21:31:18.761 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-06-27T21:31:19.074 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-06-27T21:31:19.578 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-test (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:19.713 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up radosgw (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:20.198 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-27T21:31:20.198 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-27T21:31:20.632 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-base (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:20.982 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-06-27T21:31:21.450 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mds (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:21.686 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-27T21:31:21.686 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-27T21:31:22.152 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:22.395 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-27T21:31:22.396 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-27T21:31:22.878 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-osd (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:23.171 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-27T21:31:23.171 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-27T21:31:23.671 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-k8sevents (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:23.781 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:23.907 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mon (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:24.126 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-27T21:31:24.126 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-27T21:31:24.566 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-cephadm (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:24.717 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:24.843 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-dashboard (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:25.289 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-volume (19.0.0-4578-gf51acc18-1jammy) ... 2024-06-27T21:31:25.675 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-06-27T21:31:25.776 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-06-27T21:31:25.928 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-06-27T21:31:27.071 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-06-27T21:31:27.732 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-06-27T21:31:27.740 DEBUG:teuthology.parallel:result is None 2024-06-27T21:31:27.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:31:27.962 DEBUG:teuthology.orchestra.run.smithi042:> dpkg-query -W -f '${Version}' ceph 2024-06-27T21:31:27.980 INFO:teuthology.orchestra.run.smithi042.stdout:19.0.0-4578-gf51acc18-1jammy 2024-06-27T21:31:27.981 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-4578-gf51acc18-1jammy 2024-06-27T21:31:27.981 INFO:teuthology.task.install:The correct ceph version 19.0.0-4578-gf51acc18-1jammy is installed. 2024-06-27T21:31:27.983 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-27T21:31:27.983 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-06-27T21:31:27.983 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-27T21:31:28.036 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-27T21:31:28.037 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-06-27T21:31:28.037 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/daemon-helper 2024-06-27T21:31:28.092 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-27T21:31:28.147 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-27T21:31:28.148 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-06-27T21:31:28.148 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-27T21:31:28.204 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-27T21:31:28.259 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-27T21:31:28.259 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-06-27T21:31:28.259 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/stdin-killer 2024-06-27T21:31:28.315 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-27T21:31:28.372 INFO:teuthology.run_tasks:Running task workunit... 2024-06-27T21:31:28.387 INFO:tasks.workunit:Pulling workunits from ref f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:31:28.388 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-06-27T21:31:28.388 INFO:tasks.workunit:timeout=3h 2024-06-27T21:31:28.388 INFO:tasks.workunit:cleanup=True 2024-06-27T21:31:28.388 DEBUG:teuthology.orchestra.run.smithi042:> stat -- /home/ubuntu/cephtest/mnt.0 2024-06-27T21:31:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:28.420 INFO:teuthology.orchestra.run.smithi042.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-06-27T21:31:28.421 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-06-27T21:31:28.468 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-06-27T21:31:28.468 DEBUG:teuthology.orchestra.run.smithi042:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-06-27T21:31:28.516 DEBUG:teuthology.orchestra.run.smithi042:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:31:28.566 INFO:tasks.workunit.client.0.smithi042.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-06-27T21:32:12.846 INFO:tasks.workunit.client.0.smithi042.stderr:Note: switching to 'f51acc18cc950586c82c58d666661d4391fc1122'. 2024-06-27T21:32:12.846 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-06-27T21:32:12.846 INFO:tasks.workunit.client.0.smithi042.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-06-27T21:32:12.846 INFO:tasks.workunit.client.0.smithi042.stderr:changes and commit them, and you can discard any commits you make in this 2024-06-27T21:32:12.846 INFO:tasks.workunit.client.0.smithi042.stderr:state without impacting any branches by switching back to a branch. 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr:If you want to create a new branch to retain commits you create, you may 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr:do so (now or later) by using -c with the switch command. Example: 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr: git switch -c 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr:Or undo this operation with: 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr: git switch - 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-06-27T21:32:12.847 INFO:tasks.workunit.client.0.smithi042.stderr:HEAD is now at f51acc18cc9 Merge branch 'wip-tracker-59380-admin-socket-injectfull' of https://github.com/badone/ceph into wip-yuri12-testing-2024-06-26-0904 2024-06-27T21:32:12.852 DEBUG:teuthology.orchestra.run.smithi042:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-06-27T21:32:12.903 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-06-27T21:32:12.904 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-06-27T21:32:12.953 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-06-27T21:32:12.953 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-06-27T21:32:12.953 DEBUG:teuthology.orchestra.run.smithi042:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-06-27T21:32:13.018 INFO:tasks.workunit.client.0.smithi042.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-27T21:32:13.024 INFO:tasks.workunit.client.0.smithi042.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-27T21:32:13.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T21:32:13.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T21:32:13.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-06-27T21:32:13.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-06-27T21:32:13.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-06-27T21:32:13.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-06-27T21:32:13.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-06-27T21:32:13.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-06-27T21:32:13.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/mclock-config 2024-06-27T21:32:13.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-06-27T21:32:13.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-06-27T21:32:13.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-06-27T21:32:13.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-06-27T21:32:13.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-06-27T21:32:13.025 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-06-27T21:32:13.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none ' 2024-06-27T21:32:13.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-06-27T21:32:13.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-06-27T21:32:13.026 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-06-27T21:32:13.026 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-27T21:32:13.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-06-27T21:32:13.028 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_profile_builtin_to_custom 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_profile_custom_to_builtin 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_profile_disallow_builtin_params_modify 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_profile_disallow_builtin_params_override 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_recovery_limit_adjustment_mclock' 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:32:13.029 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:32:13.030 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:32:13.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:32:13.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:32:13.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:32:13.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:32:13.032 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:32:13.032 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:32:13.033 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:32:13.034 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:32:13.034 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:32:13.034 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:32:13.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:32:13.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:32:13.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:32:13.036 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:32:13.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:32:13.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:32:13.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:32:13.039 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:32:13.039 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:13.039 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:13.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:32:13.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:32:13.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:32:13.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-27T21:32:13.042 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:32:13.042 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:13.042 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:13.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20523 2024-06-27T21:32:13.045 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:32:13.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-27T21:32:13.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-27T21:32:13.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:32:13.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-27T21:32:13.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-06-27T21:32:13.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-06-27T21:32:13.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-06-27T21:32:13.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-27T21:32:13.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-27T21:32:13.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:32:13.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:32:13.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:32:13.046 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:32:13.046 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:32:13.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:32:13.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:32:13.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:32:13.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:32:13.047 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:32:13.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:32:13.048 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:32:13.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:32:13.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:32:13.049 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:32:13.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:32:13.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:32:13.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:32:13.051 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:32:13.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:32:13.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:32:13.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:32:13.054 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:32:13.054 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:13.054 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:13.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:32:13.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:32:13.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:32:13.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-27T21:32:13.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:32:13.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:13.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:13.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20523 2024-06-27T21:32:13.059 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:32:13.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:32:13.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:32:13.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-27T21:32:13.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-06-27T21:32:13.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-27T21:32:13.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:32:13.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:32:13.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:32:13.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-27T21:32:13.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-27T21:32:13.634 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:32:13.634 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:32:13.634 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:32:13.634 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:32:13.634 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:13.634 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:13.635 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:32:13.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:32:13.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:32:13.830 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:32:13.830 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:32:13.830 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:32:13.830 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:32:13.831 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:32:13.832 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:32:13.832 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:32:13.832 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:32:13.832 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:32:13.832 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:13.832 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:13.832 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:32:13.832 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:32:13.832 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get fsid 2024-06-27T21:32:13.948 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:32:13.948 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:32:13.948 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:32:13.948 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:32:13.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:32:13.949 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:32:13.949 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:32:13.949 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:32:13.949 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:32:13.949 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:13.950 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:13.950 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:32:13.950 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:32:13.950 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get mon_host 2024-06-27T21:32:14.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-06-27T21:32:14.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-27T21:32:14.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:32:14.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:32:14.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:32:14.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-27T21:32:14.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:32:14.982 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:32:14.982 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:32:14.982 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:32:14.983 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:32:14.983 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:14.983 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:14.983 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:32:14.983 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:32:14.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:32:15.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-27T21:32:15.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-27T21:32:15.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:32:15.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:32:15.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:32:15.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-27T21:32:15.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-27T21:32:15.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:32:15.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:32:15.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:32:15.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-27T21:32:15.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-27T21:32:15.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:32:15.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:32:15.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-27T21:32:15.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:32:15.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:32:15.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:32:15.085 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:32:15.086 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:15.086 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:15.086 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:32:15.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:32:15.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:32:15.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:32:15.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:32:15.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-27T21:32:15.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-27T21:32:15.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:32:15.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:32:15.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:32:15.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:32:15.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:32:15.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-27T21:32:15.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-27T21:32:15.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:32:15.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f93faefd-7b67-40d0-b1ea-16f36967dd59 2024-06-27T21:32:15.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f93faefd-7b67-40d0-b1ea-16f36967dd59' 2024-06-27T21:32:15.088 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 f93faefd-7b67-40d0-b1ea-16f36967dd59 2024-06-27T21:32:15.088 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:32:15.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBf2n1m5LgkBhAAkOONsfHJ2agkQVfCf7Isng== 2024-06-27T21:32:15.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBf2n1m5LgkBhAAkOONsfHJ2agkQVfCf7Isng=="}' 2024-06-27T21:32:15.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f93faefd-7b67-40d0-b1ea-16f36967dd59 -i td/mclock-config/0/new.json 2024-06-27T21:32:15.436 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:32:15.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-27T21:32:15.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBf2n1m5LgkBhAAkOONsfHJ2agkQVfCf7Isng== --osd-uuid f93faefd-7b67-40d0-b1ea-16f36967dd59 2024-06-27T21:32:15.542 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:15.537+0000 7f371355b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:32:15.544 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:15.541+0000 7f371355b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:32:15.546 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:15.541+0000 7f371355b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:32:15.546 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:15.541+0000 7f371355b7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:32:15.547 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:15.541+0000 7f371355b7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:32:15.547 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:15.541+0000 7f371355b7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:32:15.596 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:15.593+0000 7f371355b7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-27T21:32:22.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-27T21:32:22.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:32:22.208 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:32:22.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:32:22.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:32:22.452 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:32:22.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:32:22.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:32:22.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:32:22.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-27T21:32:22.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:32:22.486 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:22.481+0000 7f705feed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:32:22.488 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:22.485+0000 7f705feed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:32:22.489 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:22.485+0000 7f705feed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:32:22.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:32:22.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:32:22.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:32:22.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:32:22.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:32:22.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:22.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:32:22.677 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:32:22.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:22.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:22.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:23.050 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:23.045+0000 7f705feed7c0 -1 Falling back to public interface 2024-06-27T21:32:23.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:23.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:23.901 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:32:23.902 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:32:23.902 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:23.902 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:24.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:25.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:25.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:25.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:32:25.128 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:32:25.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:25.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:25.570 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:25.781 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:32:25.777+0000 7f705feed7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:32:26.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:26.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:26.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:32:26.573 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:32:26.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:26.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:27.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:28.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:28.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:28.117 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:32:28.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:32:28.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:28.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:28.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:29.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:29.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:29.554 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:32:29.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:32:29.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:29.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:30.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:31.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:31.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:31.100 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:32:31.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:32:31.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:31.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:31.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:32.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:32.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:32.543 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:32:32.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:32:32.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:32.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:33.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:34.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:34.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:34.063 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:32:34.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:32:34.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:34.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:34.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:35.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:35.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:35.515 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:32:35.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:32:35.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:35.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:35.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:36.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:36.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:36.981 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:32:36.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:32:36.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:36.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:37.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:38.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:38.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:38.451 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:32:38.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:32:38.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:38.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:38.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:39.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:39.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:39.988 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:32:39.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:32:39.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:39.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:41.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:42.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:42.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:42.003 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:32:42.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:32:42.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:42.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:42.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:43.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:43.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:43.623 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:32:43.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:32:43.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:43.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:44.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:45.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:45.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:45.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:32:45.143 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:32:45.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:45.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:46.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:47.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:47.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:47.338 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:32:47.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:32:47.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:47.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:48.733 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:49.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:49.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:49.735 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:32:49.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:32:49.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:49.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:50.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:51.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:51.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:51.169 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T21:32:51.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T21:32:51.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:51.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:52.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:53.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:53.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:53.282 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T21:32:53.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T21:32:53.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:53.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:54.356 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:32:55.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:32:55.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:32:55.358 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T21:32:55.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T21:32:55.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:32:55.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:32:55.796 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/104594801,v1:127.0.0.1:6801/104594801] [v2:127.0.0.1:6802/104594801,v1:127.0.0.1:6803/104594801] exists,up f93faefd-7b67-40d0-b1ea-16f36967dd59 2024-06-27T21:32:55.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:32:55.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:32:55.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:32:55.797 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-06-27T21:32:55.797 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-06-27T21:32:55.799 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:32:55.799 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:32:55.799 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:32:55.799 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:32:55.799 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:55.799 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:55.799 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:32:55.800 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:32:55.800 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_max_backfills 2024-06-27T21:32:55.917 INFO:tasks.workunit.client.0.smithi042.stdout:osd_max_backfills: 1 2024-06-27T21:32:55.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-06-27T21:32:55.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-06-27T21:32:55.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-06-27T21:32:56.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-06-27T21:32:58.599 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-06-27T21:32:58.599 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-06-27T21:32:58.600 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:32:58.601 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:32:58.601 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:32:58.601 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:32:58.601 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:58.601 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:58.601 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:32:58.601 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:32:58.601 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_max_backfills 2024-06-27T21:32:58.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-06-27T21:32:58.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-06-27T21:32:58.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-06-27T21:32:58.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-06-27T21:32:58.724 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:32:58.724 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:32:58.724 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:32:58.725 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:32:58.725 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:58.725 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:58.725 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:32:58.726 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:32:58.726 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok dump_recovery_reservations 2024-06-27T21:32:58.839 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-06-27T21:32:58.839 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-06-27T21:32:58.839 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-06-27T21:32:58.839 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-06-27T21:32:58.840 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:32:58.840 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:32:58.840 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:32:58.841 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:32:58.841 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:32:58.841 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:32:58.841 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:32:58.842 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:32:58.842 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok dump_recovery_reservations 2024-06-27T21:32:58.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-06-27T21:32:58.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-06-27T21:32:58.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-06-27T21:32:59.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-06-27T21:32:59.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-06-27T21:33:01.980 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-06-27T21:33:01.980 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-06-27T21:33:01.980 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:33:01.981 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:33:01.981 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:33:01.981 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:01.981 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:01.982 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:01.982 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:33:01.982 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:33:01.982 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_max_backfills 2024-06-27T21:33:02.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-27T21:33:02.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-27T21:33:02.106 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-06-27T21:33:02.106 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-06-27T21:33:02.107 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:33:02.107 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:33:02.107 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:33:02.108 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:02.108 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:02.108 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:02.108 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:33:02.108 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:33:02.108 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok dump_recovery_reservations 2024-06-27T21:33:02.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-27T21:33:02.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-27T21:33:02.228 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-06-27T21:33:02.228 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-06-27T21:33:02.229 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:33:02.229 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:33:02.229 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:33:02.230 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:02.230 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:02.230 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:02.230 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:33:02.230 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:33:02.230 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok dump_recovery_reservations 2024-06-27T21:33:02.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-27T21:33:02.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-27T21:33:02.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-06-27T21:33:02.350 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:33:02.350 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:33:02.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:33:02.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:33:02.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:33:03.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:33:03.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-06-27T21:33:04.142 INFO:tasks.workunit.client.0.smithi042.stderr:osd.0 is already down. 2024-06-27T21:33:04.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-06-27T21:33:04.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-06-27T21:33:04.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:33:04.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:33:04.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:33:04.162 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:33:04.163 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:04.163 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:33:04.163 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:04.163 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-06-27T21:33:04.670 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6800/104594801,v1:127.0.0.1:6801/104594801] [v2:127.0.0.1:6802/104594801,v1:127.0.0.1:6803/104594801] exists f93faefd-7b67-40d0-b1ea-16f36967dd59 2024-06-27T21:33:04.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:33:04.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:33:04.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:33:04.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-06-27T21:33:04.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2024-06-27T21:33:04.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-27T21:33:04.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-27T21:33:04.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-27T21:33:04.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2024-06-27T21:33:04.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-27T21:33:04.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:33:04.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:33:04.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:33:04.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-27T21:33:04.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-27T21:33:04.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-27T21:33:04.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:33:04.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:33:04.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:33:04.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:33:04.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-27T21:33:04.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-06-27T21:33:04.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2024-06-27T21:33:04.676 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:33:04.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-27T21:33:04.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-06-27T21:33:04.677 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2024-06-27T21:33:04.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-27T21:33:04.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-27T21:33:04.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-27T21:33:04.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-27T21:33:04.706 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:04.700+0000 7f9934a227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:04.758 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:04.752+0000 7f9934a227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:04.760 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:04.756+0000 7f9934a227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:05.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-27T21:33:05.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:33:05.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:33:05.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:33:05.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:33:05.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:05.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:33:05.096 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:33:05.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:05.097 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:05.310 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:05.304+0000 7f9934a227c0 -1 Falling back to public interface 2024-06-27T21:33:05.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:06.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:06.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:06.522 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:33:06.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:33:06.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:06.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:07.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:08.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:08.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:08.026 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:33:08.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:33:08.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:08.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:08.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:08.725 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:08.720+0000 7f9934a227c0 -1 osd.0 7 log_to_monitors true 2024-06-27T21:33:09.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:09.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:09.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:33:09.454 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:33:09.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:09.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:09.932 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,7) [v2:127.0.0.1:6800/3550590276,v1:127.0.0.1:6801/3550590276] [v2:127.0.0.1:6802/3550590276,v1:127.0.0.1:6803/3550590276] exists,up f93faefd-7b67-40d0-b1ea-16f36967dd59 2024-06-27T21:33:09.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:33:09.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:33:09.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:33:09.934 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-06-27T21:33:09.935 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-06-27T21:33:09.935 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:33:09.935 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:33:09.935 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:33:09.935 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:09.935 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:09.935 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:09.935 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:33:09.935 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:33:09.935 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_max_backfills 2024-06-27T21:33:10.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-27T21:33:10.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-27T21:33:10.063 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-06-27T21:33:10.063 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-06-27T21:33:10.063 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:33:10.063 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:33:10.064 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:33:10.064 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:10.065 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:10.065 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:10.065 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:33:10.065 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:33:10.065 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok dump_recovery_reservations 2024-06-27T21:33:10.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-27T21:33:10.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-27T21:33:10.186 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-06-27T21:33:10.186 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-06-27T21:33:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:33:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:33:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:33:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:33:10.187 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:33:10.187 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok dump_recovery_reservations 2024-06-27T21:33:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-06-27T21:33:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-06-27T21:33:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-06-27T21:33:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:33:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:33:10.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:33:10.308 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:33:10.308 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:33:10.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:33:10.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:33:10.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:33:10.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:33:10.617 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:33:10.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:33:10.619 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:33:10.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:33:10.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:33:10.621 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:33:10.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:10.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:33:10.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:33:10.624 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:10.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:33:10.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:33:10.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:33:10.635 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:33:10.635 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:10.635 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:10.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:33:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:33:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:33:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-27T21:33:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:33:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:33:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:33:10.638 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:33:10.638 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:33:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:33:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:33:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:33:10.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:33:10.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:33:10.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:33:10.643 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:33:10.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:33:10.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:33:10.645 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:33:10.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:10.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:33:10.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:33:10.646 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:10.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:33:10.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:33:10.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:33:10.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:33:10.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:10.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:10.651 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:33:10.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:33:10.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:33:10.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-27T21:33:10.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-27T21:33:10.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-27T21:33:10.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-27T21:33:10.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:33:10.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:33:10.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:33:10.654 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:33:10.654 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:33:10.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:33:10.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:33:10.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:33:10.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:33:10.656 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:33:10.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:33:10.658 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:33:10.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:33:10.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:33:10.659 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:33:10.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:10.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:33:10.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:33:10.661 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:10.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:33:10.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:33:10.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:33:10.666 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:33:10.666 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:10.666 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:10.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:33:10.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:33:10.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:33:10.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-27T21:33:10.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:33:10.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:10.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:10.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20523 2024-06-27T21:33:10.671 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:33:10.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:33:10.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:33:10.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-27T21:33:10.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-06-27T21:33:10.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-06-27T21:33:10.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-06-27T21:33:10.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-27T21:33:10.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:33:10.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:33:10.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:33:10.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-27T21:33:10.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-27T21:33:11.156 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:33:11.156 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:33:11.156 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:33:11.157 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:33:11.157 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:11.157 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:11.157 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:33:11.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:33:11.384 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:33:11.384 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:33:11.385 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:33:11.385 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:33:11.385 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:33:11.386 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:33:11.386 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:33:11.386 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:33:11.386 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:33:11.386 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:11.386 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:11.386 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:11.386 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:33:11.387 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:33:11.387 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get fsid 2024-06-27T21:33:11.506 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:33:11.507 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:33:11.507 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:33:11.507 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:33:11.508 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:33:11.508 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:33:11.508 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:33:11.508 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:33:11.508 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:11.508 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:11.508 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:11.509 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:33:11.509 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:33:11.509 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get mon_host 2024-06-27T21:33:11.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-06-27T21:33:11.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-27T21:33:11.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:33:11.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:33:11.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:33:11.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-27T21:33:11.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:33:12.542 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:33:12.542 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:33:12.542 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:33:12.543 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:33:12.543 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:12.543 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:12.543 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:33:12.544 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:33:12.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:33:12.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-27T21:33:12.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-27T21:33:12.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:33:12.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:33:12.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:33:12.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-27T21:33:12.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:33:12.583 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:33:12.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-27T21:33:12.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-27T21:33:12.585 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:33:12.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0827ebae-0631-4056-858f-13f3334b0b2f 2024-06-27T21:33:12.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0827ebae-0631-4056-858f-13f3334b0b2f' 2024-06-27T21:33:12.586 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 0827ebae-0631-4056-858f-13f3334b0b2f 2024-06-27T21:33:12.586 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:33:12.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCY2n1m4OLFIxAA8uz8pFss3H6oLFU2eOpInw== 2024-06-27T21:33:12.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCY2n1m4OLFIxAA8uz8pFss3H6oLFU2eOpInw=="}' 2024-06-27T21:33:12.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0827ebae-0631-4056-858f-13f3334b0b2f -i td/mclock-config/0/new.json 2024-06-27T21:33:12.912 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:33:12.929 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-27T21:33:12.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCY2n1m4OLFIxAA8uz8pFss3H6oLFU2eOpInw== --osd-uuid 0827ebae-0631-4056-858f-13f3334b0b2f 2024-06-27T21:33:12.972 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:12.964+0000 7f0ffda8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:12.981 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:12.976+0000 7f0ffda8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:12.982 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:12.976+0000 7f0ffda8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:12.983 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:12.976+0000 7f0ffda8f7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:33:12.983 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:12.976+0000 7f0ffda8f7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:33:12.983 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:12.976+0000 7f0ffda8f7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:33:13.012 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:13.008+0000 7f0ffda8f7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-27T21:33:19.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-27T21:33:19.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:33:19.505 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:33:19.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:33:19.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:33:20.087 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:33:20.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:33:20.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-27T21:33:20.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:33:20.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:33:20.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:33:20.120 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:20.116+0000 7fdc51e327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:20.156 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:20.152+0000 7fdc51e327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:20.158 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:20.152+0000 7fdc51e327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:20.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:33:20.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:33:20.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:33:20.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:33:20.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:33:20.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:20.553 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:33:20.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:33:20.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:20.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:20.721 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:20.716+0000 7fdc51e327c0 -1 Falling back to public interface 2024-06-27T21:33:20.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:21.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:21.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:21.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:33:22.000 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:33:22.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:22.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:22.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:23.347 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:23.340+0000 7fdc51e327c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:33:23.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:23.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:23.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:33:23.432 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:33:23.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:23.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:24.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:25.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:25.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:25.057 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:33:25.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:33:25.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:25.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:25.502 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:26.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:26.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:26.505 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:33:26.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:33:26.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:26.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:26.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:27.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:27.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:27.977 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:33:27.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:33:27.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:27.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:28.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:29.489 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:29.489 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:29.489 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:33:29.490 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:33:29.490 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:29.490 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:29.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:30.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:30.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:30.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:33:30.975 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:33:30.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:30.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:31.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:32.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:32.505 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:33:32.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:32.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:33:32.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:32.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:32.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:33.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:33.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:33.959 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:33:33.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:33:33.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:33.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:34.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:35.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:35.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:35.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:33:35.560 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:33:35.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:35.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:35.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:36.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:36.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:36.996 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:33:36.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:33:36.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:36.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:37.776 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:38.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:38.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:38.778 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:33:38.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:33:38.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:38.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:39.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:40.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:40.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:40.556 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:33:40.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:33:40.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:40.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:41.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:42.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:42.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:42.378 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:33:42.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:33:42.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:42.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:43.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:44.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:44.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:44.346 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:33:44.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:33:44.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:44.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:45.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:46.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:46.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:46.165 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:33:46.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:33:46.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:46.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:47.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:33:48.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:33:48.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:33:48.183 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:33:48.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:33:48.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:33:48.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:33:48.616 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4140823555,v1:127.0.0.1:6803/4140823555] [v2:127.0.0.1:6804/4140823555,v1:127.0.0.1:6805/4140823555] exists,up 0827ebae-0631-4056-858f-13f3334b0b2f 2024-06-27T21:33:48.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:33:48.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:33:48.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:33:48.617 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-06-27T21:33:49.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-06-27T21:33:49.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-06-27T21:33:49.108 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-06-27T21:33:49.109 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-06-27T21:33:49.109 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:33:49.109 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:33:49.109 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:49.109 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:49.109 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:49.110 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:33:49.110 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-06-27T21:33:49.110 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_profile 2024-06-27T21:33:49.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-06-27T21:33:49.234 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-06-27T21:33:49.234 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-06-27T21:33:49.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-06-27T21:33:49.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-06-27T21:33:49.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-06-27T21:33:49.387 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:33:49.387 INFO:tasks.workunit.client.0.smithi042.stdout: "success": "" 2024-06-27T21:33:49.387 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:33:49.398 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-06-27T21:33:49.399 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-06-27T21:33:49.399 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:33:49.399 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:33:49.399 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:49.399 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:49.399 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:49.399 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:33:49.399 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-06-27T21:33:49.400 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_profile 2024-06-27T21:33:49.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-06-27T21:33:49.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-06-27T21:33:49.517 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-06-27T21:33:49.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-06-27T21:33:49.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-06-27T21:33:49.519 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-06-27T21:33:49.519 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-06-27T21:33:49.519 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:33:49.519 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:33:49.519 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-06-27T21:33:49.519 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:49.519 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:49.519 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:49.520 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:33:49.520 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-06-27T21:33:49.520 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-27T21:33:49.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-06-27T21:33:49.641 INFO:tasks.workunit.client.0.smithi042.stdout:client_res = 0.300000 2024-06-27T21:33:49.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-06-27T21:33:49.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-06-27T21:33:49.642 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-06-27T21:33:49.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-06-27T21:33:49.644 INFO:tasks.workunit.client.0.smithi042.stdout:client_res_new = .400000 2024-06-27T21:33:49.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-06-27T21:33:49.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-06-27T21:33:50.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-27T21:33:50.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-06-27T21:33:50.562 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-06-27T21:33:50.562 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-06-27T21:33:50.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-06-27T21:33:50.566 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-06-27T21:33:50.566 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-06-27T21:33:50.566 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-06-27T21:33:50.566 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:33:50.566 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:33:50.566 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:50.566 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:50.566 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:50.566 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:33:50.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-06-27T21:33:50.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-27T21:33:50.689 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-06-27T21:33:50.690 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-06-27T21:33:50.690 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-06-27T21:33:50.692 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-06-27T21:33:50.692 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-06-27T21:33:50.692 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:33:50.692 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:33:50.693 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:33:50.693 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:33:50.693 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:33:50.693 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:33:50.693 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:33:50.693 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:33:50.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:33:50.801 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:33:50.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:33:50.802 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:33:50.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:33:50.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:33:50.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:33:50.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:50.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:33:50.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:33:50.804 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:50.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:33:50.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:33:50.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:33:50.811 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:33:50.811 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:50.812 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:50.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:33:50.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:33:50.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:33:50.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-27T21:33:50.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:33:50.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:33:50.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:33:50.815 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:33:50.815 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:33:50.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:33:50.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:33:50.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:33:50.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:33:50.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:33:50.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:33:50.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:33:50.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:33:50.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:33:50.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:33:50.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:50.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:33:50.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:33:50.819 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:50.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:33:50.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:33:50.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:33:50.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:33:50.822 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:50.822 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:50.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:33:50.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:33:50.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:33:50.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-27T21:33:50.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-27T21:33:50.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-27T21:33:50.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-27T21:33:50.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:33:50.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:33:50.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:33:50.824 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:33:50.824 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:33:50.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:33:50.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:33:50.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:33:50.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:33:50.825 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:33:50.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:33:50.826 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:33:50.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:33:50.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:33:50.827 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:33:50.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:50.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:33:50.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:33:50.828 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:50.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:33:50.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:33:50.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:33:50.831 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:33:50.831 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:50.831 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:50.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:33:50.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:33:50.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:33:50.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-27T21:33:50.833 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:33:50.833 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:50.833 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:50.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20523 2024-06-27T21:33:50.835 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:33:50.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:33:50.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:33:50.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:33:50.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:33:50.837 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:33:50.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:33:50.838 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:33:50.839 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:33:50.839 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:33:50.839 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:33:50.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:50.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:33:50.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:33:50.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:33:50.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:33:50.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:33:50.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:33:50.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:33:50.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:50.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:50.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:33:50.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:33:50.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:33:50.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-27T21:33:50.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:33:50.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:50.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:50.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20523 2024-06-27T21:33:50.847 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:33:50.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:33:50.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:33:50.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-27T21:33:50.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-06-27T21:33:50.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-27T21:33:50.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:33:50.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:33:50.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:33:50.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-27T21:33:50.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-27T21:33:51.343 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:33:51.407 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:33:51.408 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:33:51.408 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:33:51.409 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:51.409 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:51.409 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:33:51.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:33:51.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:33:51.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:33:51.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:33:51.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:33:51.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:33:51.566 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:33:51.566 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:33:51.566 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:33:51.566 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:33:51.567 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:51.567 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:51.567 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:51.567 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:33:51.568 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:33:51.568 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get fsid 2024-06-27T21:33:51.686 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:33:51.686 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:33:51.687 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:33:51.687 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:33:51.688 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:33:51.688 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:33:51.688 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:33:51.688 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:33:51.688 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:33:51.688 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:51.688 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:51.689 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:33:51.689 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:33:51.689 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get mon_host 2024-06-27T21:33:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-06-27T21:33:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-27T21:33:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:33:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:33:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:33:51.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-27T21:33:51.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:33:52.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:33:52.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:33:52.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:33:52.737 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:33:52.737 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:52.737 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:52.737 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:33:52.738 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:33:52.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:33:52.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-27T21:33:52.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:33:52.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:33:52.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-27T21:33:52.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:33:52.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:33:52.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:33:52.815 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:33:52.815 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:33:52.815 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:33:52.815 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:33:52.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:33:52.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:33:52.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:33:52.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:33:52.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-27T21:33:52.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-27T21:33:52.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:33:52.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:33:52.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:33:52.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:33:52.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:33:52.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-27T21:33:52.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-27T21:33:52.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:33:52.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b6f99a3f-2760-43a0-96fa-3baf614355a9 2024-06-27T21:33:52.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b6f99a3f-2760-43a0-96fa-3baf614355a9' 2024-06-27T21:33:52.817 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 b6f99a3f-2760-43a0-96fa-3baf614355a9 2024-06-27T21:33:52.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:33:52.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDA2n1mtGi2MRAA+XVk7Y321Mg26gAScdinuA== 2024-06-27T21:33:52.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDA2n1mtGi2MRAA+XVk7Y321Mg26gAScdinuA=="}' 2024-06-27T21:33:52.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b6f99a3f-2760-43a0-96fa-3baf614355a9 -i td/mclock-config/0/new.json 2024-06-27T21:33:53.116 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:33:53.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-27T21:33:53.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDA2n1mtGi2MRAA+XVk7Y321Mg26gAScdinuA== --osd-uuid b6f99a3f-2760-43a0-96fa-3baf614355a9 2024-06-27T21:33:53.166 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:53.160+0000 7fb570fc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:53.184 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:53.180+0000 7fb570fc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:53.185 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:53.180+0000 7fb570fc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:33:53.186 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:53.180+0000 7fb570fc87c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:33:53.186 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:53.180+0000 7fb570fc87c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:33:53.186 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:53.180+0000 7fb570fc87c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:33:53.224 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:33:53.220+0000 7fb570fc87c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-27T21:33:59.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-27T21:33:59.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:33:59.921 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:33:59.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:33:59.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:34:00.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:34:00.508 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:34:00.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-27T21:34:00.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:34:00.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:34:00.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:34:00.542 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:00.536+0000 7efec4f5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:34:00.566 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:00.560+0000 7efec4f5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:34:00.568 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:00.560+0000 7efec4f5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:34:00.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:34:00.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:34:00.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:34:00.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:34:00.982 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:34:00.983 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:34:00.983 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:00.983 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:34:00.983 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:00.983 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:01.157 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:01.152+0000 7efec4f5f7c0 -1 Falling back to public interface 2024-06-27T21:34:01.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:02.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:02.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:02.450 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:34:02.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:34:02.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:02.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:02.929 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:03.791 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:03.784+0000 7efec4f5f7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:34:03.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:03.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:03.931 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:34:03.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:34:03.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:03.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:04.489 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:05.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:05.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:05.491 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:34:05.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:34:05.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:05.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:05.984 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:06.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:06.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:06.990 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:34:06.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:34:06.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:06.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:07.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:08.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:08.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:08.436 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:34:08.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:34:08.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:08.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:08.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:09.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:09.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:09.885 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:34:09.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:34:09.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:09.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:10.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:11.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:11.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:11.345 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:34:11.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:34:11.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:11.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:11.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:12.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:12.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:12.800 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:34:12.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:34:12.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:12.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:13.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:14.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:14.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:14.244 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:34:14.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:34:14.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:14.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:14.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:15.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:15.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:15.786 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:34:15.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:34:15.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:15.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:16.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:17.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:17.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:17.234 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:34:17.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:34:17.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:17.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:18.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:19.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:19.165 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:34:19.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:19.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:34:19.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:19.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:19.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:20.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:20.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:20.989 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:34:20.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:34:20.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:20.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:21.746 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:22.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:22.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:22.748 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:34:22.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:34:22.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:22.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:23.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:24.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:24.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:24.527 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:34:24.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:34:24.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:24.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:25.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:26.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:26.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:26.736 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:34:26.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:34:26.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:26.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:27.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:28.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:28.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:28.605 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:34:28.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:34:28.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:28.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:29.057 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/229386992,v1:127.0.0.1:6803/229386992] [v2:127.0.0.1:6804/229386992,v1:127.0.0.1:6805/229386992] exists,up b6f99a3f-2760-43a0-96fa-3baf614355a9 2024-06-27T21:34:29.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:34:29.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:34:29.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:34:29.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-06-27T21:34:29.058 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-06-27T21:34:29.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-06-27T21:34:29.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-06-27T21:34:29.528 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-06-27T21:34:29.528 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-27T21:34:29.528 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:34:29.528 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:34:29.528 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:34:29.528 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:29.528 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:29.529 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:34:29.529 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-27T21:34:29.529 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_profile 2024-06-27T21:34:29.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-06-27T21:34:29.647 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-06-27T21:34:29.647 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-06-27T21:34:29.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-06-27T21:34:29.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-06-27T21:34:29.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-06-27T21:34:29.796 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:34:29.797 INFO:tasks.workunit.client.0.smithi042.stdout: "success": "" 2024-06-27T21:34:29.797 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:34:29.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-06-27T21:34:29.814 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-27T21:34:29.814 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:34:29.814 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:34:29.814 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:34:29.814 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:29.814 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:29.814 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:34:29.814 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-27T21:34:29.815 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_profile 2024-06-27T21:34:29.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-06-27T21:34:29.934 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-06-27T21:34:29.934 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-06-27T21:34:29.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-06-27T21:34:29.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-06-27T21:34:29.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-06-27T21:34:29.936 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-27T21:34:29.936 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-06-27T21:34:29.936 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:34:29.936 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-06-27T21:34:29.936 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:34:29.937 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:34:29.937 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:29.937 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:29.937 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:34:29.937 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-27T21:34:29.937 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-27T21:34:30.048 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-06-27T21:34:30.049 INFO:tasks.workunit.client.0.smithi042.stdout:Original client_res for osd.0 = 0.300000 2024-06-27T21:34:30.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-06-27T21:34:30.049 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-06-27T21:34:30.050 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-06-27T21:34:30.050 INFO:tasks.workunit.client.0.smithi042.stdout:client_res_new = .400000 2024-06-27T21:34:30.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-06-27T21:34:30.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-06-27T21:34:30.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-06-27T21:34:30.550 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-27T21:34:31.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-06-27T21:34:31.001 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-06-27T21:34:31.001 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-06-27T21:34:31.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-27T21:34:31.005 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-06-27T21:34:31.005 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-27T21:34:31.005 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:34:31.005 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:34:31.005 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-06-27T21:34:31.006 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:34:31.006 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:31.006 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:31.006 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:34:31.006 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-27T21:34:31.006 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-27T21:34:31.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-06-27T21:34:31.122 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-06-27T21:34:31.123 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-06-27T21:34:31.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-27T21:34:31.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-06-27T21:34:31.274 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:34:31.274 INFO:tasks.workunit.client.0.smithi042.stdout: "success": "" 2024-06-27T21:34:31.274 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:34:31.290 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-06-27T21:34:31.290 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-27T21:34:31.290 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:34:31.290 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:34:31.290 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:34:31.291 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:31.291 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:31.291 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:34:31.291 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-27T21:34:31.291 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_profile 2024-06-27T21:34:31.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-06-27T21:34:31.409 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-06-27T21:34:31.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-06-27T21:34:31.409 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-27T21:34:31.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-06-27T21:34:31.842 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-06-27T21:34:31.842 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-06-27T21:34:31.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-27T21:34:31.846 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-06-27T21:34:31.846 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-06-27T21:34:31.846 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-27T21:34:31.846 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:34:31.846 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:34:31.846 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:34:31.846 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:31.846 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:31.846 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:34:31.847 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-27T21:34:31.847 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-27T21:34:31.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-06-27T21:34:31.973 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-06-27T21:34:31.973 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-06-27T21:34:31.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-27T21:34:31.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-06-27T21:34:32.481 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-06-27T21:34:37.483 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-27T21:34:37.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-06-27T21:34:37.950 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-06-27T21:34:37.950 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-06-27T21:34:37.951 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-27T21:34:37.953 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-06-27T21:34:37.953 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-06-27T21:34:37.953 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-06-27T21:34:37.953 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:34:37.953 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:34:37.953 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:34:37.953 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:37.953 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:37.954 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:34:37.954 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-06-27T21:34:37.954 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-27T21:34:38.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-06-27T21:34:38.074 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-06-27T21:34:38.075 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-06-27T21:34:38.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-06-27T21:34:38.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-06-27T21:34:38.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:34:38.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:34:38.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:34:38.078 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:34:38.078 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:34:38.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:34:38.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:34:38.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:34:38.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:34:38.185 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:34:38.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:34:38.186 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:34:38.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:34:38.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:34:38.188 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:34:38.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:34:38.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:34:38.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:34:38.189 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:34:38.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:34:38.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:34:38.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:34:38.196 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:34:38.196 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:38.196 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:38.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:34:38.197 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:34:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:34:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-27T21:34:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:34:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:34:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:34:38.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:34:38.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:34:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:34:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:34:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:34:38.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:34:38.199 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:34:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:34:38.200 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:34:38.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:34:38.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:34:38.202 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:34:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:34:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:34:38.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:34:38.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:34:38.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:34:38.204 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:34:38.204 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:34:38.206 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:34:38.207 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:38.207 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:38.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:34:38.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:34:38.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:34:38.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-27T21:34:38.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-27T21:34:38.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-27T21:34:38.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-27T21:34:38.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:34:38.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:34:38.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:34:38.209 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:34:38.209 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:34:38.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:34:38.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:34:38.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:34:38.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:34:38.210 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:34:38.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:34:38.211 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:34:38.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:34:38.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:34:38.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:34:38.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:34:38.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:34:38.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:34:38.213 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:34:38.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:34:38.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:34:38.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:34:38.216 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:34:38.216 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:38.216 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:38.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:34:38.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:34:38.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:34:38.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-27T21:34:38.218 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:34:38.218 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:38.218 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:38.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20523 2024-06-27T21:34:38.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:34:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:34:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:34:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-27T21:34:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-06-27T21:34:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-06-27T21:34:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-06-27T21:34:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-27T21:34:38.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-27T21:34:38.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:34:38.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:34:38.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:34:38.222 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:34:38.222 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:34:38.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:34:38.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:34:38.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:34:38.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:34:38.223 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:34:38.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:34:38.223 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:34:38.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:34:38.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:34:38.224 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:34:38.225 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:34:38.225 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:34:38.225 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:34:38.226 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:34:38.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:34:38.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:34:38.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:34:38.228 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:34:38.228 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:38.228 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:38.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:34:38.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:34:38.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:34:38.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-27T21:34:38.230 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:34:38.231 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:38.231 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:38.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20523 2024-06-27T21:34:38.232 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:34:38.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:34:38.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:34:38.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-27T21:34:38.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-06-27T21:34:38.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-27T21:34:38.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:34:38.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:34:38.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:34:38.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-27T21:34:38.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-27T21:34:38.708 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:34:38.708 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:34:38.708 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:34:38.708 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:34:38.708 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:38.708 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:38.708 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:34:38.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:34:38.902 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:34:38.903 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:34:38.903 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:34:38.903 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:34:38.903 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:34:38.903 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:34:38.904 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:34:38.904 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:34:38.904 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:34:38.905 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:34:38.905 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:38.905 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:38.905 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:34:38.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:34:38.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get fsid 2024-06-27T21:34:39.025 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:34:39.025 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:34:39.025 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:34:39.025 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:34:39.025 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:34:39.026 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:34:39.026 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:34:39.026 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:34:39.026 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:34:39.026 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:39.027 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:39.027 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:34:39.027 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:34:39.027 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get mon_host 2024-06-27T21:34:39.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-06-27T21:34:39.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-27T21:34:39.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:34:39.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:34:39.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:34:39.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-27T21:34:39.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:34:40.049 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:34:40.049 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:34:40.049 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:34:40.049 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:34:40.049 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:40.049 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:40.049 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:34:40.050 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:34:40.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:34:40.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-27T21:34:40.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-27T21:34:40.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:34:40.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:34:40.092 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:34:40.093 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:34:40.093 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:34:40.093 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:34:40.093 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:34:40.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:34:40.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:34:40.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:34:40.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:34:40.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-27T21:34:40.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-27T21:34:40.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:34:40.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:34:40.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:34:40.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:34:40.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:34:40.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-27T21:34:40.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-27T21:34:40.095 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:34:40.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e4114c5b-1c7f-4778-a3ed-27d8cd3327e2 2024-06-27T21:34:40.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e4114c5b-1c7f-4778-a3ed-27d8cd3327e2' 2024-06-27T21:34:40.095 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 e4114c5b-1c7f-4778-a3ed-27d8cd3327e2 2024-06-27T21:34:40.096 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:34:40.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDw2n1mFsGQBhAACWGnC+dnx69DF0FCXDYkqg== 2024-06-27T21:34:40.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDw2n1mFsGQBhAACWGnC+dnx69DF0FCXDYkqg=="}' 2024-06-27T21:34:40.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e4114c5b-1c7f-4778-a3ed-27d8cd3327e2 -i td/mclock-config/0/new.json 2024-06-27T21:34:40.456 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:34:40.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-27T21:34:40.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDw2n1mFsGQBhAACWGnC+dnx69DF0FCXDYkqg== --osd-uuid e4114c5b-1c7f-4778-a3ed-27d8cd3327e2 2024-06-27T21:34:40.515 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:40.512+0000 7f12177e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:34:40.533 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:40.532+0000 7f12177e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:34:40.534 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:40.532+0000 7f12177e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:34:40.535 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:40.532+0000 7f12177e97c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:34:40.535 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:40.532+0000 7f12177e97c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:34:40.535 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:40.532+0000 7f12177e97c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:34:40.573 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:40.572+0000 7f12177e97c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-27T21:34:47.133 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-27T21:34:47.133 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:34:47.135 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:34:47.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:34:47.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:34:47.735 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:34:47.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:34:47.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-27T21:34:47.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:34:47.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:34:47.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:34:47.767 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:47.764+0000 7fd9a12577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:34:47.772 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:47.772+0000 7fd9a12577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:34:47.773 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:47.772+0000 7fd9a12577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:34:48.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:34:48.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:34:48.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:34:48.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:34:48.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:34:48.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:48.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:34:48.168 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:34:48.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:48.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:48.354 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:48.352+0000 7fd9a12577c0 -1 Falling back to public interface 2024-06-27T21:34:48.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:49.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:49.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:49.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:34:49.636 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:34:49.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:49.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:50.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:50.930 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:34:50.928+0000 7fd9a12577c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:34:51.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:51.072 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:34:51.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:51.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:34:51.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:51.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:51.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:52.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:52.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:52.585 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:34:52.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:34:52.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:52.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:53.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:54.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:54.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:54.064 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:34:54.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:34:54.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:54.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:54.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:55.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:55.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:55.556 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:34:55.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:34:55.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:55.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:56.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:57.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:57.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:57.141 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:34:57.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:34:57.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:57.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:57.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:34:58.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:34:58.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:34:58.681 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:34:58.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:34:58.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:34:58.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:34:59.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:00.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:00.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:00.119 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:35:00.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:35:00.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:00.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:00.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:01.699 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:01.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:01.700 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:35:01.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:35:01.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:01.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:02.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:03.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:03.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:03.167 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:35:03.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:35:03.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:03.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:03.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:04.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:04.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:04.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:35:04.911 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:35:04.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:04.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:06.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:07.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:07.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:07.037 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:35:07.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:35:07.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:07.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:07.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:08.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:08.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:08.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:35:08.779 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:35:08.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:08.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:09.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:10.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:10.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:10.603 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:35:10.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:35:10.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:10.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:11.318 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:12.319 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:12.319 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:12.320 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:35:12.320 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:35:12.320 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:12.320 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:13.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:14.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:14.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:14.607 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:35:14.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:35:14.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:14.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:15.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:16.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:16.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:16.076 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:35:16.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:35:16.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:16.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:16.505 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1091196454,v1:127.0.0.1:6803/1091196454] [v2:127.0.0.1:6804/1091196454,v1:127.0.0.1:6805/1091196454] exists,up e4114c5b-1c7f-4778-a3ed-27d8cd3327e2 2024-06-27T21:35:16.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:35:16.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:35:16.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:35:16.506 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-06-27T21:35:17.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-06-27T21:35:17.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-06-27T21:35:17.001 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-06-27T21:35:17.001 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-06-27T21:35:17.001 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:35:17.002 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:35:17.002 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:35:17.002 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:17.002 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:17.003 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:35:17.003 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-06-27T21:35:17.003 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_profile 2024-06-27T21:35:17.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-06-27T21:35:17.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-06-27T21:35:17.119 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-06-27T21:35:17.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-06-27T21:35:17.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-06-27T21:35:17.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-06-27T21:35:17.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-06-27T21:35:17.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-06-27T21:35:17.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-06-27T21:35:17.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-06-27T21:35:17.122 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-06-27T21:35:17.122 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-06-27T21:35:17.122 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-06-27T21:35:17.122 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:35:17.122 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:35:17.123 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:35:17.123 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:17.123 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:17.123 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:35:17.123 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-06-27T21:35:17.124 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-06-27T21:35:17.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-06-27T21:35:17.245 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-06-27T21:35:17.245 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:17.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-06-27T21:35:17.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-06-27T21:35:17.917 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-06-27T21:35:17.918 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-06-27T21:35:17.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-06-27T21:35:17.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-06-27T21:35:17.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-06-27T21:35:19.971 INFO:tasks.workunit.client.0.smithi042.stdout:Check configuration values - Attempt#: 0 2024-06-27T21:35:19.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-06-27T21:35:19.972 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-06-27T21:35:20.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-06-27T21:35:20.410 INFO:tasks.workunit.client.0.smithi042.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-06-27T21:35:20.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-06-27T21:35:20.411 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-06-27T21:35:20.411 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:20.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-06-27T21:35:20.414 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-06-27T21:35:20.414 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-06-27T21:35:20.414 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-06-27T21:35:20.415 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-06-27T21:35:20.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-06-27T21:35:20.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-06-27T21:35:20.838 INFO:tasks.workunit.client.0.smithi042.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-06-27T21:35:20.838 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-06-27T21:35:20.838 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:20.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:20.841 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-06-27T21:35:20.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-06-27T21:35:20.848 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-06-27T21:35:20.848 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-06-27T21:35:20.848 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-06-27T21:35:20.848 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:35:20.849 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:35:20.849 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:35:20.849 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:20.849 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:20.849 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:35:20.849 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-06-27T21:35:20.849 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-06-27T21:35:20.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-06-27T21:35:20.959 INFO:tasks.workunit.client.0.smithi042.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-06-27T21:35:20.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-06-27T21:35:20.960 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-06-27T21:35:20.960 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:20.962 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-06-27T21:35:20.962 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:20.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-06-27T21:35:20.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-06-27T21:35:20.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-06-27T21:35:20.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-06-27T21:35:20.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-06-27T21:35:20.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-06-27T21:35:20.965 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-06-27T21:35:20.965 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:35:20.965 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:35:20.966 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:35:20.966 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:20.966 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:20.966 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:35:20.966 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-06-27T21:35:20.966 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-27T21:35:21.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-06-27T21:35:21.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-06-27T21:35:21.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:21.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-06-27T21:35:21.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-06-27T21:35:21.719 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-06-27T21:35:21.720 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-06-27T21:35:21.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-06-27T21:35:21.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-06-27T21:35:21.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-06-27T21:35:23.724 INFO:tasks.workunit.client.0.smithi042.stdout:Check configuration values - Attempt#: 0 2024-06-27T21:35:23.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-06-27T21:35:23.724 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-27T21:35:24.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-06-27T21:35:24.161 INFO:tasks.workunit.client.0.smithi042.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-06-27T21:35:24.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-06-27T21:35:24.162 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-06-27T21:35:24.162 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:24.163 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-06-27T21:35:24.164 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-06-27T21:35:24.164 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-06-27T21:35:24.164 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-06-27T21:35:24.164 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-06-27T21:35:24.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-06-27T21:35:24.598 INFO:tasks.workunit.client.0.smithi042.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-06-27T21:35:24.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-06-27T21:35:24.599 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-06-27T21:35:24.599 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:24.602 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:24.602 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-06-27T21:35:24.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-06-27T21:35:24.604 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-06-27T21:35:24.604 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-06-27T21:35:24.604 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-06-27T21:35:24.604 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:35:24.604 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:35:24.605 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:35:24.605 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:24.605 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:24.605 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:35:24.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-06-27T21:35:24.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-27T21:35:24.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-06-27T21:35:24.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-06-27T21:35:24.717 INFO:tasks.workunit.client.0.smithi042.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-06-27T21:35:24.718 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-06-27T21:35:24.718 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:24.720 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-06-27T21:35:24.720 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-06-27T21:35:24.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-06-27T21:35:24.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-06-27T21:35:24.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-06-27T21:35:24.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-06-27T21:35:24.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:35:24.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:35:24.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:35:24.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:35:24.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:35:24.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:35:24.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:35:24.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:35:24.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:35:24.831 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:35:24.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:35:24.832 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:35:24.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:35:24.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:35:24.834 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:35:24.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:35:24.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:35:24.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:35:24.835 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:35:24.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:35:24.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:35:24.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:35:24.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:35:24.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:24.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:24.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:35:24.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:35:24.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:35:24.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-27T21:35:24.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:35:24.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:35:24.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:35:24.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:35:24.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:35:24.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:35:24.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:35:24.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:35:24.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:35:24.848 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:35:24.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:35:24.849 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:35:24.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:35:24.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:35:24.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:35:24.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:35:24.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:35:24.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:35:24.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:35:24.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:35:24.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:35:24.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:35:24.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:35:24.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:24.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:24.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:35:24.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:35:24.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:35:24.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-27T21:35:24.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-27T21:35:24.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-27T21:35:24.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-27T21:35:24.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:35:24.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:35:24.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:35:24.858 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:35:24.858 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:35:24.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:35:24.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:35:24.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:35:24.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:35:24.859 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:35:24.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:35:24.860 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:35:24.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:35:24.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:35:24.862 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:35:24.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:35:24.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:35:24.863 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:35:24.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:35:24.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:35:24.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:35:24.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:35:24.866 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:35:24.866 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:24.866 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:24.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:35:24.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:35:24.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:35:24.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-27T21:35:24.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:35:24.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:24.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:24.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20523 2024-06-27T21:35:24.871 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:35:24.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:35:24.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:35:24.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-27T21:35:24.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-06-27T21:35:24.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-06-27T21:35:24.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-06-27T21:35:24.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-27T21:35:24.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-27T21:35:24.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:35:24.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:35:24.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:35:24.872 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:35:24.872 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:35:24.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:35:24.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:35:24.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:35:24.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:35:24.873 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:35:24.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:35:24.874 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:35:24.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:35:24.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:35:24.876 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:35:24.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:35:24.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:35:24.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:35:24.877 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:35:24.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:35:24.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:35:24.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:35:24.880 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:35:24.880 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:24.880 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:24.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:35:24.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:35:24.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:35:24.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-27T21:35:24.883 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:35:24.883 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:24.883 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:24.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20523 2024-06-27T21:35:24.885 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:35:24.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:35:24.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:35:24.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-27T21:35:24.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-06-27T21:35:24.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-27T21:35:24.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:35:24.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:35:24.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:35:24.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-27T21:35:24.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-27T21:35:25.435 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:35:25.435 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:35:25.435 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:35:25.435 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:35:25.436 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:25.436 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:25.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:35:25.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:35:25.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:35:25.654 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:35:25.654 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:35:25.654 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:35:25.654 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:35:25.655 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:35:25.655 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:35:25.655 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:35:25.655 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:35:25.656 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:35:25.656 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:25.656 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:25.656 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:35:25.656 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:35:25.656 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get fsid 2024-06-27T21:35:25.775 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:35:25.775 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:35:25.775 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:35:25.775 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:35:25.776 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:35:25.776 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:35:25.776 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:35:25.776 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:35:25.777 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:35:25.777 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:25.777 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:25.777 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:35:25.777 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:35:25.777 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get mon_host 2024-06-27T21:35:25.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-06-27T21:35:25.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-27T21:35:25.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:35:25.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:35:25.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:35:25.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-27T21:35:25.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:35:26.786 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:35:26.786 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:35:26.786 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:35:26.787 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:35:26.787 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:26.787 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:26.787 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:35:26.787 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:35:26.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:35:26.826 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-27T21:35:26.827 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:35:26.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:35:26.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:35:26.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:35:26.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-27T21:35:26.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-27T21:35:26.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:35:26.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=338f1669-08bf-429c-924a-3a4c8e56fcb8 2024-06-27T21:35:26.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 338f1669-08bf-429c-924a-3a4c8e56fcb8' 2024-06-27T21:35:26.830 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 338f1669-08bf-429c-924a-3a4c8e56fcb8 2024-06-27T21:35:26.830 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:35:26.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAe231mun4+MhAAU9SRCkCMcbAoIfcgKQrCkw== 2024-06-27T21:35:26.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAe231mun4+MhAAU9SRCkCMcbAoIfcgKQrCkw=="}' 2024-06-27T21:35:26.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 338f1669-08bf-429c-924a-3a4c8e56fcb8 -i td/mclock-config/0/new.json 2024-06-27T21:35:27.126 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:35:27.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-27T21:35:27.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAe231mun4+MhAAU9SRCkCMcbAoIfcgKQrCkw== --osd-uuid 338f1669-08bf-429c-924a-3a4c8e56fcb8 2024-06-27T21:35:27.178 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:27.175+0000 7fb5412e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:35:27.195 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:27.191+0000 7fb5412e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:35:27.195 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:27.191+0000 7fb5412e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:35:27.195 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:27.191+0000 7fb5412e27c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:35:27.195 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:27.191+0000 7fb5412e27c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:35:27.195 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:27.191+0000 7fb5412e27c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:35:27.232 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:27.231+0000 7fb5412e27c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-27T21:35:33.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-27T21:35:33.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:35:33.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:35:33.857 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:35:33.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:35:34.462 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:35:34.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:35:34.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:35:34.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-27T21:35:34.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:35:34.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:35:34.494 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:34.491+0000 7f3436c5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:35:34.500 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:34.495+0000 7f3436c5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:35:34.501 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:34.499+0000 7f3436c5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:35:34.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:35:34.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:35:34.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:35:34.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:35:34.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:35:34.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:34.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:35:34.887 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:35:34.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:34.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:35.098 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:35.095+0000 7f3436c5d7c0 -1 Falling back to public interface 2024-06-27T21:35:35.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:36.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:36.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:36.382 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:35:36.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:35:36.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:36.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:36.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:37.634 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:35:37.631+0000 7f3436c5d7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:35:37.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:37.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:37.820 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:35:37.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:35:37.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:37.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:38.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:39.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:39.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:39.432 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:35:39.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:35:39.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:39.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:39.884 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:40.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:40.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:40.886 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:35:40.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:35:40.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:40.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:41.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:42.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:42.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:42.423 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:35:42.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:35:42.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:42.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:42.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:43.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:43.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:43.882 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:35:43.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:35:43.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:43.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:44.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:45.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:45.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:45.413 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:35:45.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:35:45.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:45.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:45.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:46.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:46.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:46.974 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:35:46.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:35:46.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:46.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:47.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:48.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:48.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:48.515 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:35:48.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:35:48.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:48.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:48.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:49.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:49.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:49.964 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:35:49.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:35:49.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:49.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:50.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:51.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:51.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:51.990 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:35:51.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:35:51.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:51.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:52.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:53.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:53.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:53.921 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:35:53.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:35:53.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:53.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:54.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:55.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:55.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:55.713 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:35:55.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:35:55.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:55.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:56.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:57.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:57.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:57.200 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:35:57.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:35:57.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:57.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:35:58.318 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:35:59.321 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:35:59.321 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:35:59.321 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:35:59.321 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:35:59.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:35:59.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:00.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:01.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:01.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:01.063 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:36:01.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:36:01.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:01.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:01.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:02.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:02.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:02.660 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:36:02.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:36:02.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:02.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:03.119 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2513216629,v1:127.0.0.1:6803/2513216629] [v2:127.0.0.1:6804/2513216629,v1:127.0.0.1:6805/2513216629] exists,up 338f1669-08bf-429c-924a-3a4c8e56fcb8 2024-06-27T21:36:03.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:36:03.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:36:03.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:36:03.120 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-06-27T21:36:03.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-06-27T21:36:03.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-06-27T21:36:03.565 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-06-27T21:36:03.565 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:36:03.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-06-27T21:36:03.565 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:36:03.565 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:03.565 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:03.565 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:03.565 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:36:03.566 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-06-27T21:36:03.566 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_profile 2024-06-27T21:36:03.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-06-27T21:36:03.693 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-06-27T21:36:03.693 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-06-27T21:36:03.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-06-27T21:36:03.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-06-27T21:36:03.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-06-27T21:36:03.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-06-27T21:36:03.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-06-27T21:36:03.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-06-27T21:36:03.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-06-27T21:36:03.695 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-06-27T21:36:03.695 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-06-27T21:36:03.695 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-06-27T21:36:03.695 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:36:03.695 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:36:03.696 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:03.697 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:03.697 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:03.697 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:36:03.697 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-06-27T21:36:03.697 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-06-27T21:36:03.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-06-27T21:36:03.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-06-27T21:36:03.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:03.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-06-27T21:36:03.826 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-06-27T21:36:03.970 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:36:03.970 INFO:tasks.workunit.client.0.smithi042.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2024-06-27T21:36:03.970 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:36:03.985 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-06-27T21:36:03.986 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-06-27T21:36:03.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-06-27T21:36:03.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-06-27T21:36:03.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-06-27T21:36:05.988 INFO:tasks.workunit.client.0.smithi042.stdout:Check configuration values - Attempt#: 0 2024-06-27T21:36:05.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-06-27T21:36:05.989 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-06-27T21:36:06.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-06-27T21:36:06.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-06-27T21:36:06.431 INFO:tasks.workunit.client.0.smithi042.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-06-27T21:36:06.432 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-06-27T21:36:06.432 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:06.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-06-27T21:36:06.433 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-06-27T21:36:06.434 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-06-27T21:36:06.434 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-06-27T21:36:06.434 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-06-27T21:36:06.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-06-27T21:36:06.864 INFO:tasks.workunit.client.0.smithi042.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-06-27T21:36:06.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-06-27T21:36:06.864 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-06-27T21:36:06.865 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:06.866 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-06-27T21:36:06.866 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:06.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-06-27T21:36:06.869 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-06-27T21:36:06.869 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-06-27T21:36:06.869 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-06-27T21:36:06.869 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:36:06.869 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:36:06.870 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:06.870 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:06.870 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:06.870 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:36:06.870 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-06-27T21:36:06.870 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-06-27T21:36:06.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-06-27T21:36:06.991 INFO:tasks.workunit.client.0.smithi042.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-06-27T21:36:06.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-06-27T21:36:06.992 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:06.992 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-06-27T21:36:06.994 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-06-27T21:36:06.994 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:06.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-06-27T21:36:06.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-06-27T21:36:06.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-06-27T21:36:06.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-06-27T21:36:06.996 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-06-27T21:36:06.997 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-06-27T21:36:06.997 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-06-27T21:36:06.997 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:36:06.997 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:36:06.998 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:06.998 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:06.998 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:06.998 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:36:06.998 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-06-27T21:36:06.998 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-27T21:36:07.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-06-27T21:36:07.124 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:07.124 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-06-27T21:36:07.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-06-27T21:36:07.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-06-27T21:36:07.263 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:36:07.263 INFO:tasks.workunit.client.0.smithi042.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2024-06-27T21:36:07.263 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:36:07.277 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-06-27T21:36:07.278 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-06-27T21:36:07.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-06-27T21:36:07.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-06-27T21:36:07.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-06-27T21:36:09.280 INFO:tasks.workunit.client.0.smithi042.stdout:Check configuration values - Attempt#: 0 2024-06-27T21:36:09.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-06-27T21:36:09.281 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-06-27T21:36:09.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-06-27T21:36:09.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-06-27T21:36:09.731 INFO:tasks.workunit.client.0.smithi042.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-06-27T21:36:09.731 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-06-27T21:36:09.732 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:09.733 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-06-27T21:36:09.735 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-06-27T21:36:09.735 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-06-27T21:36:09.735 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-06-27T21:36:09.735 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-06-27T21:36:10.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-06-27T21:36:10.169 INFO:tasks.workunit.client.0.smithi042.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-06-27T21:36:10.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-06-27T21:36:10.170 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:10.170 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-06-27T21:36:10.172 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-06-27T21:36:10.172 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:10.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-06-27T21:36:10.179 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-06-27T21:36:10.179 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-06-27T21:36:10.179 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-06-27T21:36:10.179 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:36:10.179 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:36:10.179 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:10.180 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:10.180 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:10.180 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:36:10.180 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-06-27T21:36:10.180 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-06-27T21:36:10.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-06-27T21:36:10.300 INFO:tasks.workunit.client.0.smithi042.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-06-27T21:36:10.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-06-27T21:36:10.301 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-06-27T21:36:10.301 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:10.303 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-06-27T21:36:10.303 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-06-27T21:36:10.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-06-27T21:36:10.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-06-27T21:36:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-06-27T21:36:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-06-27T21:36:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:36:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:36:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:36:10.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:36:10.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:36:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:36:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:36:10.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:36:10.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:36:10.413 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:36:10.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:36:10.415 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:36:10.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:36:10.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:36:10.416 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:36:10.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:10.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:36:10.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:36:10.420 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:10.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:36:10.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:36:10.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:36:10.425 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:36:10.425 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:10.425 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:10.425 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:36:10.426 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:36:10.426 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:36:10.427 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-27T21:36:10.427 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:36:10.427 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:36:10.427 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:36:10.427 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:36:10.427 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:36:10.427 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:36:10.428 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:36:10.428 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:36:10.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:36:10.429 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:36:10.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:36:10.430 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:36:10.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:36:10.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:36:10.432 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:36:10.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:10.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:36:10.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:36:10.433 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:10.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:36:10.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:36:10.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:36:10.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:36:10.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:10.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:10.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:36:10.438 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:36:10.438 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:36:10.438 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-06-27T21:36:10.438 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-06-27T21:36:10.438 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-27T21:36:10.438 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-27T21:36:10.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:36:10.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:36:10.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:36:10.439 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:36:10.439 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:36:10.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:36:10.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:36:10.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:36:10.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:36:10.440 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:36:10.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:36:10.441 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:36:10.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:36:10.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:36:10.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:36:10.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:10.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:36:10.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:36:10.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:10.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:36:10.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:36:10.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:36:10.446 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:36:10.446 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:10.447 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:10.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:36:10.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:36:10.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:36:10.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-27T21:36:10.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:36:10.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:10.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:10.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20523 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:36:10.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:36:10.453 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:36:10.453 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:36:10.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:36:10.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:36:10.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:36:10.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:36:10.454 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:36:10.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:36:10.455 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:36:10.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:36:10.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:36:10.456 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:36:10.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:10.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:36:10.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:36:10.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:10.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:36:10.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:36:10.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:36:10.460 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:36:10.460 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:10.460 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:10.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:36:10.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:36:10.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:36:10.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-06-27T21:36:10.463 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:36:10.463 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:10.463 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:10.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20523 2024-06-27T21:36:10.465 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:36:10.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:36:10.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:36:10.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-06-27T21:36:10.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-06-27T21:36:10.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-06-27T21:36:10.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:36:10.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:36:10.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:36:10.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-06-27T21:36:10.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-06-27T21:36:10.955 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:36:10.956 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:36:10.956 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:36:10.956 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:36:10.956 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:10.956 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:10.957 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:36:10.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:36:11.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:36:11.176 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:36:11.176 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:36:11.176 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:36:11.177 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:36:11.178 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:36:11.178 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:36:11.179 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:36:11.179 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:36:11.179 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:11.179 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:11.179 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:11.179 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:36:11.179 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:36:11.179 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get fsid 2024-06-27T21:36:11.296 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:36:11.296 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:36:11.297 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:36:11.297 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:36:11.298 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:36:11.298 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:36:11.298 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:36:11.298 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:36:11.298 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:11.298 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:11.298 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:11.298 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-mon.a.asok 2024-06-27T21:36:11.299 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:36:11.299 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20523/ceph-mon.a.asok config get mon_host 2024-06-27T21:36:11.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-06-27T21:36:11.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-06-27T21:36:11.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:36:11.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:36:11.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:36:11.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-06-27T21:36:11.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:36:12.334 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:36:12.334 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:36:12.334 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:36:12.334 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:36:12.334 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:12.334 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:12.334 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:36:12.335 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:36:12.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:36:12.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-06-27T21:36:12.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:36:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:36:12.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:36:12.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-06-27T21:36:12.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-06-27T21:36:12.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:36:12.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:36:12.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:36:12.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:36:12.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:36:12.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-27T21:36:12.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-06-27T21:36:12.374 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:36:12.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cfd3f299-5cf4-4325-bc31-f8b1f1f6740a 2024-06-27T21:36:12.375 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 cfd3f299-5cf4-4325-bc31-f8b1f1f6740a 2024-06-27T21:36:12.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cfd3f299-5cf4-4325-bc31-f8b1f1f6740a' 2024-06-27T21:36:12.375 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:36:12.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBM231maZwgFxAACJDUumjQhndAYpTth9U3uA== 2024-06-27T21:36:12.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBM231maZwgFxAACJDUumjQhndAYpTth9U3uA=="}' 2024-06-27T21:36:12.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cfd3f299-5cf4-4325-bc31-f8b1f1f6740a -i td/mclock-config/0/new.json 2024-06-27T21:36:12.738 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:36:12.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-06-27T21:36:12.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBM231maZwgFxAACJDUumjQhndAYpTth9U3uA== --osd-uuid cfd3f299-5cf4-4325-bc31-f8b1f1f6740a 2024-06-27T21:36:12.782 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:12.779+0000 7fc157cf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:36:12.823 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:12.819+0000 7fc157cf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:36:12.825 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:12.823+0000 7fc157cf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:36:12.825 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:12.823+0000 7fc157cf57c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:36:12.825 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:12.823+0000 7fc157cf57c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:36:12.825 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:12.823+0000 7fc157cf57c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-06-27T21:36:12.863 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:12.859+0000 7fc157cf57c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-06-27T21:36:19.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-06-27T21:36:19.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:36:19.460 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:36:19.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:36:19.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:36:20.062 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:36:20.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:36:20.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e141cb2f-ce8d-4d94-ab2d-0020094ea649 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.20523/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-27T21:36:20.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:36:20.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:36:20.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:36:20.094 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:20.091+0000 7f3b3d87b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:36:20.114 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:20.111+0000 7f3b3d87b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:36:20.115 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:20.111+0000 7f3b3d87b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:36:20.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:36:20.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:36:20.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:36:20.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:36:20.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:36:20.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:20.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:36:20.529 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:36:20.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:20.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:20.694 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:20.691+0000 7f3b3d87b7c0 -1 Falling back to public interface 2024-06-27T21:36:20.984 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:21.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:21.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:21.987 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:36:21.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:36:21.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:21.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:22.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:23.331 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:23.327+0000 7f3b3d87b7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:36:23.462 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:23.462 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:23.463 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:36:23.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:36:23.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:23.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:24.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:25.032 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:25.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:25.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:36:25.033 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:36:25.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:25.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:25.496 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:26.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:26.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:26.499 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:36:26.499 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:36:26.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:26.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:26.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:27.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:27.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:27.997 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:36:27.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:36:27.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:27.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:28.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:29.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:29.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:29.505 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:36:29.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:36:29.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:29.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:29.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:30.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:31.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:31.000 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:36:31.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:36:31.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:31.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:31.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:32.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:32.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:32.552 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:36:32.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:36:32.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:32.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:33.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:34.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:34.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:34.020 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:36:34.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:36:34.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:34.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:34.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:35.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:35.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:35.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:36:35.494 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:36:35.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:35.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:35.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:36.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:36.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:36.935 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:36:36.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:36:36.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:36.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:37.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:38.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:38.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:38.874 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:36:38.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:36:38.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:38.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:39.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:40.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:40.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:40.524 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:36:40.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:36:40.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:40.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:41.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:42.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:42.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:42.382 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:36:42.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:36:42.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:42.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:43.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:44.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:44.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:44.265 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:36:44.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:36:44.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:44.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:45.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:46.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:46.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:46.043 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:36:46.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:36:46.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:46.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:47.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:36:48.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:36:48.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:36:48.327 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:36:48.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:36:48.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:36:48.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:36:48.862 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1759903524,v1:127.0.0.1:6803/1759903524] [v2:127.0.0.1:6804/1759903524,v1:127.0.0.1:6805/1759903524] exists,up cfd3f299-5cf4-4325-bc31-f8b1f1f6740a 2024-06-27T21:36:48.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:36:48.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:36:48.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:36:48.864 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:36:48.864 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:36:48.864 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:36:48.865 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:48.865 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:48.865 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:48.866 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:36:48.866 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:36:48.866 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_recovery_max_active 2024-06-27T21:36:48.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-06-27T21:36:48.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-06-27T21:36:48.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-06-27T21:36:48.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-06-27T21:36:49.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-06-27T21:36:51.719 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:36:51.719 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:36:51.719 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:36:51.720 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:51.720 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:51.720 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:51.720 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:36:51.720 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:36:51.720 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_recovery_max_active 2024-06-27T21:36:51.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-06-27T21:36:51.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-06-27T21:36:51.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-06-27T21:36:52.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-06-27T21:36:52.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-06-27T21:36:54.947 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-06-27T21:36:54.948 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:36:54.948 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:36:54.948 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:54.949 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:54.949 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:54.949 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20523/ceph-osd.0.asok 2024-06-27T21:36:54.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-06-27T21:36:54.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.20523/ceph-osd.0.asok config get osd_recovery_max_active 2024-06-27T21:36:55.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-06-27T21:36:55.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-06-27T21:36:55.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-06-27T21:36:55.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:36:55.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:36:55.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:36:55.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:36:55.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:36:55.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:36:55.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:36:55.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:36:55.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:36:55.195 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:36:55.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:36:55.196 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:36:55.197 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:36:55.197 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:36:55.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:36:55.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:55.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:36:55.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:36:55.199 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:55.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:36:55.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:36:55.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:36:55.207 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:36:55.207 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:55.207 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:55.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:36:55.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:36:55.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:36:55.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-06-27T21:36:55.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:36:55.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:36:55.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:36:55.209 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:36:55.209 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:36:55.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:36:55.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:36:55.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:36:55.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:36:55.211 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:36:55.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:36:55.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:36:55.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:36:55.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:36:55.213 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:36:55.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:55.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:36:55.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:36:55.215 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:55.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:36:55.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:36:55.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:36:55.218 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:36:55.218 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:55.218 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:55.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/mclock-config 0 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:36:55.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:36:55.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:36:55.222 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:36:55.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:36:55.223 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:36:55.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:36:55.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:36:55.225 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:36:55.225 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:55.225 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:36:55.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:36:55.226 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:55.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:36:55.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-27T21:36:55.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-06-27T21:36:55.229 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:36:55.229 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:55.229 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20523 2024-06-27T21:36:55.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20523 2024-06-27T21:36:55.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:36:55.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:36:55.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-06-27T21:36:55.232 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-27T21:36:55.232 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-27T21:36:55.284 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-06-27T21:36:55.284 DEBUG:teuthology.orchestra.run.smithi042:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-06-27T21:36:55.343 INFO:tasks.workunit.client.0.smithi042.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-27T21:36:55.348 INFO:tasks.workunit.client.0.smithi042.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-27T21:36:55.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T21:36:55.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T21:36:55.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-06-27T21:36:55.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-06-27T21:36:55.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-06-27T21:36:55.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-06-27T21:36:55.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-06-27T21:36:55.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-06-27T21:36:55.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/network-ping 2024-06-27T21:36:55.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-06-27T21:36:55.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-06-27T21:36:55.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-06-27T21:36:55.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-06-27T21:36:55.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-06-27T21:36:55.349 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-06-27T21:36:55.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none ' 2024-06-27T21:36:55.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-06-27T21:36:55.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-06-27T21:36:55.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-06-27T21:36:55.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-06-27T21:36:55.350 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-06-27T21:36:55.350 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-27T21:36:55.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-06-27T21:36:55.352 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_network_ping_test2' 2024-06-27T21:36:55.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-06-27T21:36:55.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-06-27T21:36:55.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-06-27T21:36:55.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-06-27T21:36:55.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-06-27T21:36:55.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:36:55.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-06-27T21:36:55.353 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:36:55.353 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:36:55.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:36:55.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:36:55.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:36:55.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:36:55.354 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:36:55.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:36:55.355 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:36:55.356 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:36:55.356 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:36:55.356 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:36:55.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:55.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:36:55.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:36:55.358 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:36:55.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:36:55.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:36:55.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-06-27T21:36:55.361 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:36:55.361 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:55.361 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:36:55.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34572 2024-06-27T21:36:55.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:36:55.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:36:55.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-06-27T21:36:55.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:36:55.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:55.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:36:55.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.34572 2024-06-27T21:36:55.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:36:55.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-06-27T21:36:55.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-06-27T21:36:55.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:36:55.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:36:55.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:36:55.841 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:36:55.841 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:55.841 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:36:55.841 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:36:55.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:36:56.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:36:56.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:36:56.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:36:56.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:36:56.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:36:56.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:36:56.085 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:36:56.085 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:36:56.085 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:36:56.086 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:56.086 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:56.086 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:36:56.086 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-mon.a.asok 2024-06-27T21:36:56.086 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:36:56.086 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34572/ceph-mon.a.asok config get fsid 2024-06-27T21:36:56.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:36:56.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:36:56.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:36:56.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:36:56.204 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:36:56.204 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:36:56.204 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:36:56.205 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:36:56.205 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:36:56.205 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:56.205 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:36:56.205 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-mon.a.asok 2024-06-27T21:36:56.205 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:36:56.206 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34572/ceph-mon.a.asok config get mon_host 2024-06-27T21:36:56.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-06-27T21:36:56.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-06-27T21:36:56.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:36:56.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:36:56.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:36:56.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-06-27T21:36:56.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:36:57.241 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:36:57.241 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:36:57.241 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:36:57.241 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:36:57.241 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:57.241 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:36:57.241 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:36:57.242 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:36:57.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:36:57.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:36:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-27T21:36:57.309 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:36:57.309 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:36:57.309 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:36:57.310 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:36:57.310 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:36:57.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-06-27T21:36:57.312 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:36:57.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff8d02a0-d344-4356-820b-9524ca98f411 2024-06-27T21:36:57.312 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 ff8d02a0-d344-4356-820b-9524ca98f411 2024-06-27T21:36:57.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ff8d02a0-d344-4356-820b-9524ca98f411' 2024-06-27T21:36:57.313 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:36:57.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5231mOLO7ExAAXumzQNG3w1TRMcXIcyww3w== 2024-06-27T21:36:57.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5231mOLO7ExAAXumzQNG3w1TRMcXIcyww3w=="}' 2024-06-27T21:36:57.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff8d02a0-d344-4356-820b-9524ca98f411 -i td/network-ping/0/new.json 2024-06-27T21:36:57.630 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:36:57.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-06-27T21:36:57.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5231mOLO7ExAAXumzQNG3w1TRMcXIcyww3w== --osd-uuid ff8d02a0-d344-4356-820b-9524ca98f411 2024-06-27T21:36:57.683 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:57.678+0000 7fe489d7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:36:57.698 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:57.694+0000 7fe489d7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:36:57.699 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:57.694+0000 7fe489d7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:36:57.700 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:57.694+0000 7fe489d7d7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-27T21:36:57.700 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:57.694+0000 7fe489d7d7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-27T21:36:57.700 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:57.694+0000 7fe489d7d7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-27T21:36:57.738 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:36:57.734+0000 7fe489d7d7c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-06-27T21:37:04.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-06-27T21:37:04.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:37:04.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:37:04.281 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:37:04.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:37:04.887 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:37:04.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:37:04.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:37:04.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:37:04.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:37:04.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:37:04.922 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:04.918+0000 7f90d0f517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:37:04.924 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:04.918+0000 7f90d0f517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:37:04.925 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:04.922+0000 7f90d0f517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:37:05.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:37:05.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:37:05.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:37:05.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:37:05.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:37:05.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:05.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:37:05.324 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:37:05.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:05.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:05.499 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:05.494+0000 7f90d0f517c0 -1 Falling back to public interface 2024-06-27T21:37:05.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:06.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:06.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:06.768 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:37:06.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:37:06.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:06.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:07.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:08.078 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:08.074+0000 7f90d0f517c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:37:08.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:08.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:08.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:37:08.208 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:37:08.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:08.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:08.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:09.753 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:09.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:09.754 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:37:09.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:37:09.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:09.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:10.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:11.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:11.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:11.210 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:37:11.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:37:11.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:11.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:11.652 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:12.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:12.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:12.654 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:37:12.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:37:12.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:12.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:13.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:14.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:14.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:14.172 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:37:14.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:37:14.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:14.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:14.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:15.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:15.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:15.615 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:37:15.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:37:15.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:15.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:16.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:17.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:17.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:17.059 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:37:17.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:37:17.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:17.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:17.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:18.536 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:18.536 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:18.537 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:37:18.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:37:18.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:18.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:19.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:20.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:20.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:20.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:37:20.030 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:37:20.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:20.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:20.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:21.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:21.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:21.515 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:37:21.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:37:21.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:21.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:22.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:23.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:23.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:23.458 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:37:23.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:37:23.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:23.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:24.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:25.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:25.330 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:25.330 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:37:25.331 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:37:25.331 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:25.331 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:25.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:26.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:26.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:26.957 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:37:26.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:37:26.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:26.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:28.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:29.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:29.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:29.094 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:37:29.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:37:29.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:29.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:29.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:30.968 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:30.968 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:30.968 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:37:30.968 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:37:30.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:30.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:31.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:32.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:32.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:32.866 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:37:32.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:37:32.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:32.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:37:33.335 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3845049258,v1:127.0.0.1:6803/3845049258] [v2:127.0.0.1:6804/3845049258,v1:127.0.0.1:6805/3845049258] exists,up ff8d02a0-d344-4356-820b-9524ca98f411 2024-06-27T21:37:33.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:37:33.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:37:33.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:37:33.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-06-27T21:37:33.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-27T21:37:33.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:37:33.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T21:37:33.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:37:33.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-06-27T21:37:33.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-27T21:37:33.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:37:33.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:37:33.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:37:33.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-06-27T21:37:33.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-06-27T21:37:33.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:37:33.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:37:33.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-27T21:37:33.337 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:37:33.337 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:37:33.337 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:37:33.338 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:37:33.338 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:37:33.338 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:37:33.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-06-27T21:37:33.340 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:37:33.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3c413121-a26d-420c-990c-3ad11591c69b 2024-06-27T21:37:33.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3c413121-a26d-420c-990c-3ad11591c69b' 2024-06-27T21:37:33.341 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 3c413121-a26d-420c-990c-3ad11591c69b 2024-06-27T21:37:33.342 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:37:33.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCd231mNf68FRAAdmB/nXJtd2mrPhoKsVlX8w== 2024-06-27T21:37:33.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCd231mNf68FRAAdmB/nXJtd2mrPhoKsVlX8w=="}' 2024-06-27T21:37:33.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3c413121-a26d-420c-990c-3ad11591c69b -i td/network-ping/1/new.json 2024-06-27T21:37:33.999 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:37:34.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-06-27T21:37:34.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCd231mNf68FRAAdmB/nXJtd2mrPhoKsVlX8w== --osd-uuid 3c413121-a26d-420c-990c-3ad11591c69b 2024-06-27T21:37:34.057 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:34.054+0000 7f47fdbf07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:37:34.059 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:34.054+0000 7f47fdbf07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:37:34.060 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:34.054+0000 7f47fdbf07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:37:34.061 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:34.058+0000 7f47fdbf07c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-27T21:37:34.061 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:34.058+0000 7f47fdbf07c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-27T21:37:34.061 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:34.058+0000 7f47fdbf07c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-27T21:37:34.101 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:34.098+0000 7f47fdbf07c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-06-27T21:37:40.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-06-27T21:37:40.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:37:40.491 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T21:37:40.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T21:37:40.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:37:41.085 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T21:37:41.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T21:37:41.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:37:41.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:37:41.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:37:41.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:37:41.119 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:41.114+0000 7f86ca0c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:37:41.143 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:41.138+0000 7f86ca0c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:37:41.144 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:41.138+0000 7f86ca0c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:37:41.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T21:37:41.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:37:41.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T21:37:41.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:37:41.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:37:41.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:41.565 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:37:41.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:37:41.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:41.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:41.731 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:41.726+0000 7f86ca0c17c0 -1 Falling back to public interface 2024-06-27T21:37:42.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:43.032 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:43.032 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:43.032 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:37:43.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:37:43.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:43.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:43.490 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:44.474 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:37:44.470+0000 7f86ca0c17c0 -1 osd.1 0 log_to_monitors true 2024-06-27T21:37:44.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:44.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:44.491 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:37:44.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:37:44.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:44.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:45.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:46.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:46.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:46.071 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:37:46.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:37:46.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:46.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:46.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:47.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:47.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:47.624 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:37:47.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:37:47.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:47.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:48.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:49.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:49.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:49.075 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:37:49.075 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:37:49.075 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:49.075 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:49.580 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:50.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:50.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:50.582 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:37:50.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:37:50.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:50.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:51.106 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:52.108 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:52.108 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:52.109 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:37:52.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:37:52.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:52.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:52.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:53.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:53.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:53.899 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:37:53.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:37:53.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:53.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:54.481 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:55.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:55.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:55.484 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:37:55.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:37:55.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:55.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:55.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:56.965 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:56.965 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:56.965 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:37:56.966 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:37:56.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:56.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:57.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:37:58.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:37:58.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:37:58.548 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:37:58.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:37:58.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:37:58.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:37:59.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:00.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:00.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:00.811 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:38:00.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:38:00.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:00.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:38:01.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:02.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:02.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:02.677 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:38:02.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:38:02.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:02.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:38:04.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:05.002 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:05.002 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:05.002 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:38:05.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:38:05.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:05.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:38:05.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:06.753 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:06.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:06.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:38:06.754 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:38:06.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:06.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:38:07.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:08.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:08.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:08.866 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:38:08.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:38:08.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:08.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:38:09.382 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2604016113,v1:127.0.0.1:6811/2604016113] [v2:127.0.0.1:6812/2604016113,v1:127.0.0.1:6813/2604016113] exists,up 3c413121-a26d-420c-990c-3ad11591c69b 2024-06-27T21:38:09.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:38:09.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:38:09.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:38:09.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-06-27T21:38:09.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-27T21:38:09.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:38:09.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-27T21:38:09.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:38:09.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-06-27T21:38:09.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-27T21:38:09.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:38:09.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:38:09.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:38:09.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-06-27T21:38:09.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-06-27T21:38:09.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:38:09.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:38:09.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-27T21:38:09.387 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:38:09.387 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:38:09.387 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:38:09.387 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:38:09.387 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:38:09.387 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:38:09.387 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:38:09.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c6a7a7e5-d814-4678-b7c2-fe66dc6393e1 2024-06-27T21:38:09.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c6a7a7e5-d814-4678-b7c2-fe66dc6393e1' 2024-06-27T21:38:09.390 INFO:tasks.workunit.client.0.smithi042.stdout:add osd2 c6a7a7e5-d814-4678-b7c2-fe66dc6393e1 2024-06-27T21:38:09.390 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:38:09.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDB231mx6mPGBAARnZzKwD8Z/SmaT7iwDfNXQ== 2024-06-27T21:38:09.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDB231mx6mPGBAARnZzKwD8Z/SmaT7iwDfNXQ=="}' 2024-06-27T21:38:09.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c6a7a7e5-d814-4678-b7c2-fe66dc6393e1 -i td/network-ping/2/new.json 2024-06-27T21:38:09.909 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:38:09.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-06-27T21:38:09.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDB231mx6mPGBAARnZzKwD8Z/SmaT7iwDfNXQ== --osd-uuid c6a7a7e5-d814-4678-b7c2-fe66dc6393e1 2024-06-27T21:38:09.962 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:09.958+0000 7f5ce5e847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:38:09.988 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:09.982+0000 7f5ce5e847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:38:09.989 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:09.986+0000 7f5ce5e847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:38:09.989 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:09.986+0000 7f5ce5e847c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-27T21:38:09.990 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:09.986+0000 7f5ce5e847c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-27T21:38:09.990 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:09.986+0000 7f5ce5e847c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-27T21:38:10.027 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:10.022+0000 7f5ce5e847c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-06-27T21:38:16.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-06-27T21:38:16.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:38:16.709 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd2 key to auth repository 2024-06-27T21:38:16.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-27T21:38:16.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:38:17.297 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T21:38:17.298 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-27T21:38:17.298 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:38:17.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:38:17.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:38:17.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:38:17.333 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:17.326+0000 7f3576b8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:38:17.354 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:17.350+0000 7f3576b8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:38:17.355 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:17.350+0000 7f3576b8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:38:17.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-27T21:38:17.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:38:17.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T21:38:17.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:38:17.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:38:17.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:17.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:38:17.745 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:38:17.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:17.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:17.930 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:17.926+0000 7f3576b8c7c0 -1 Falling back to public interface 2024-06-27T21:38:18.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:19.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:19.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:19.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:38:19.190 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:38:19.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:19.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:19.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:20.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:20.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:20.635 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:38:20.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:38:20.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:20.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:20.698 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:38:20.694+0000 7f3576b8c7c0 -1 osd.2 0 log_to_monitors true 2024-06-27T21:38:21.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:22.197 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:22.197 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:22.197 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:38:22.197 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:38:22.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:22.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:22.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:23.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:23.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:23.944 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:38:23.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:38:23.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:23.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:24.580 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:25.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:25.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:25.582 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:38:25.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:38:25.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:25.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:26.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:27.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:27.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:27.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:38:27.029 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:38:27.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:27.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:27.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:28.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:28.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:28.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:38:28.624 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:38:28.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:28.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:29.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:30.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:30.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:30.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:38:30.245 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:38:30.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:30.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:30.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:31.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:31.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:31.977 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:38:31.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:38:31.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:31.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:32.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:33.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:33.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:33.544 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:38:33.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:38:33.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:33.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:33.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:34.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:34.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:34.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:38:34.997 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:38:34.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:34.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:35.890 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:36.892 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:36.892 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:36.892 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:38:36.892 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:38:36.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:36.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:37.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:38.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:38.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:38.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:38:38.763 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:38:38.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:38.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:39.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:40.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:40.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:40.543 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:38:40.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:38:40.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:40.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:41.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:42.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:42.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:42.687 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:38:42.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:38:42.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:42.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:43.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:44.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:44.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:44.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:38:44.551 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:38:44.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:44.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:45.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:38:46.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:38:46.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:38:46.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:38:46.423 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:38:46.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:38:46.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:38:46.922 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2653659508,v1:127.0.0.1:6819/2653659508] [v2:127.0.0.1:6820/2653659508,v1:127.0.0.1:6821/2653659508] exists,up c6a7a7e5-d814-4678-b7c2-fe66dc6393e1 2024-06-27T21:38:46.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:38:46.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:38:46.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:38:46.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-06-27T21:38:51.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-06-27T21:38:51.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-06-27T21:38:52.506 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'foo' already exists 2024-06-27T21:38:52.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T21:38:53.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-06-27T21:38:53.568 INFO:tasks.workunit.client.0.smithi042.stdout:hints = 1 2024-06-27T21:38:53.568 INFO:tasks.workunit.client.0.smithi042.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-06-27T21:38:53.568 INFO:tasks.workunit.client.0.smithi042.stdout:Object prefix: benchmark_data_smithi042_40036 2024-06-27T21:38:53.568 INFO:tasks.workunit.client.0.smithi042.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:38:53.568 INFO:tasks.workunit.client.0.smithi042.stdout: 0 0 0 0 0 0 - 0 2024-06-27T21:38:54.568 INFO:tasks.workunit.client.0.smithi042.stdout: 1 16 59 43 0.167954 0.167969 0.169013 0.27128 2024-06-27T21:38:55.568 INFO:tasks.workunit.client.0.smithi042.stdout: 2 16 115 99 0.193335 0.21875 0.329145 0.317067 2024-06-27T21:38:56.568 INFO:tasks.workunit.client.0.smithi042.stdout: 3 16 163 147 0.191381 0.1875 0.279134 0.31366 2024-06-27T21:38:57.568 INFO:tasks.workunit.client.0.smithi042.stdout: 4 16 227 211 0.206027 0.25 0.287685 0.300053 2024-06-27T21:38:58.568 INFO:tasks.workunit.client.0.smithi042.stdout: 5 16 276 260 0.203098 0.191406 0.283972 0.29383 2024-06-27T21:38:59.568 INFO:tasks.workunit.client.0.smithi042.stdout: 6 16 340 324 0.210908 0.25 0.284137 0.288818 2024-06-27T21:39:00.569 INFO:tasks.workunit.client.0.smithi042.stdout: 7 16 403 387 0.215929 0.246094 0.290533 0.284954 2024-06-27T21:39:01.569 INFO:tasks.workunit.client.0.smithi042.stdout: 8 16 452 436 0.21286 0.191406 0.267208 0.28792 2024-06-27T21:39:02.569 INFO:tasks.workunit.client.0.smithi042.stdout: 9 16 515 499 0.216548 0.246094 0.287058 0.284899 2024-06-27T21:39:03.569 INFO:tasks.workunit.client.0.smithi042.stdout: 10 16 564 548 0.214031 0.191406 0.274204 0.285299 2024-06-27T21:39:04.569 INFO:tasks.workunit.client.0.smithi042.stdout:Total time run: 10.1168 2024-06-27T21:39:04.570 INFO:tasks.workunit.client.0.smithi042.stdout:Total writes made: 564 2024-06-27T21:39:04.570 INFO:tasks.workunit.client.0.smithi042.stdout:Write size: 4096 2024-06-27T21:39:04.570 INFO:tasks.workunit.client.0.smithi042.stdout:Object size: 4096 2024-06-27T21:39:04.570 INFO:tasks.workunit.client.0.smithi042.stdout:Bandwidth (MB/sec): 0.217769 2024-06-27T21:39:04.570 INFO:tasks.workunit.client.0.smithi042.stdout:Stddev Bandwidth: 0.0316703 2024-06-27T21:39:04.570 INFO:tasks.workunit.client.0.smithi042.stdout:Max bandwidth (MB/sec): 0.25 2024-06-27T21:39:04.570 INFO:tasks.workunit.client.0.smithi042.stdout:Min bandwidth (MB/sec): 0.167969 2024-06-27T21:39:04.570 INFO:tasks.workunit.client.0.smithi042.stdout:Average IOPS: 55 2024-06-27T21:39:04.570 INFO:tasks.workunit.client.0.smithi042.stdout:Stddev IOPS: 8.10761 2024-06-27T21:39:04.571 INFO:tasks.workunit.client.0.smithi042.stdout:Max IOPS: 64 2024-06-27T21:39:04.571 INFO:tasks.workunit.client.0.smithi042.stdout:Min IOPS: 43 2024-06-27T21:39:04.571 INFO:tasks.workunit.client.0.smithi042.stdout:Average Latency(s): 0.284838 2024-06-27T21:39:04.571 INFO:tasks.workunit.client.0.smithi042.stdout:Stddev Latency(s): 0.0458445 2024-06-27T21:39:04.571 INFO:tasks.workunit.client.0.smithi042.stdout:Max latency(s): 0.484803 2024-06-27T21:39:04.571 INFO:tasks.workunit.client.0.smithi042.stdout:Min latency(s): 0.169013 2024-06-27T21:39:04.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-06-27T21:39:14.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-06-27T21:39:14.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:39:14.653 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:39:15.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T21:39:15.095 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T21:39:15.095 INFO:tasks.workunit.client.0.smithi042.stderr:2' 2024-06-27T21:39:15.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:39:15.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:39:15.096 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:39:15.258 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836506 2024-06-27T21:39:15.258 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836506 2024-06-27T21:39:15.258 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506' 2024-06-27T21:39:15.258 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:39:15.259 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T21:39:15.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672979 2024-06-27T21:39:15.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672979 2024-06-27T21:39:15.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672979' 2024-06-27T21:39:15.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:39:15.418 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T21:39:15.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509451 2024-06-27T21:39:15.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509451 2024-06-27T21:39:15.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672979 2-64424509451' 2024-06-27T21:39:15.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:39:15.566 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836506 2024-06-27T21:39:15.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:39:15.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:39:15.568 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-06-27T21:39:15.568 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:39:15.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836506 2024-06-27T21:39:15.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-06-27T21:39:15.570 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836506 2024-06-27T21:39:15.570 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:39:15.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836506 2024-06-27T21:39:15.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T21:39:16.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T21:39:16.983 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:39:17.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836506 2024-06-27T21:39:17.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:39:17.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672979 2024-06-27T21:39:17.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:39:17.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T21:39:17.459 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672979 2024-06-27T21:39:17.460 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:39:17.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672979 2024-06-27T21:39:17.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-06-27T21:39:17.461 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672979 2024-06-27T21:39:17.462 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T21:39:17.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672979 2024-06-27T21:39:17.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:39:17.950 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509451 2024-06-27T21:39:17.950 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:39:17.951 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T21:39:17.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509451 2024-06-27T21:39:17.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:39:17.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509451 2024-06-27T21:39:17.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-06-27T21:39:17.954 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509451 2024-06-27T21:39:17.954 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T21:39:18.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509452 -lt 64424509451 2024-06-27T21:39:18.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:39:18.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-06-27T21:39:18.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:39:18.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:39:18.453 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:39:18.453 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:39:18.454 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:39:18.454 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-osd.0.asok 2024-06-27T21:39:18.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:39:18.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-osd.0.asok dump_osd_network 2024-06-27T21:39:18.562 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:39:18.562 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 1000, 2024-06-27T21:39:18.562 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [] 2024-06-27T21:39:18.562 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:39:18.577 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:18.577 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:39:18.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:39:18.614 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-06-27T21:39:18.614 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:18.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-06-27T21:39:18.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:39:18.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-27T21:39:18.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-27T21:39:18.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-27T21:39:18.652 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:39:18.652 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:39:18.652 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:39:18.652 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-mgr.x.asok 2024-06-27T21:39:18.652 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:39:18.652 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-mgr.x.asok dump_osd_network 2024-06-27T21:39:18.752 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:39:18.752 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 1000, 2024-06-27T21:39:18.752 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [] 2024-06-27T21:39:18.752 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:39:18.766 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:18.766 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:39:18.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:39:18.805 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:18.805 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-06-27T21:39:18.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-06-27T21:39:18.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:39:18.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-06-27T21:39:18.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:39:18.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:39:18.844 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:39:18.844 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:39:18.844 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:39:18.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-osd.0.asok 2024-06-27T21:39:18.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:39:18.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-osd.0.asok dump_osd_network 0 2024-06-27T21:39:18.952 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:39:18.952 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 0, 2024-06-27T21:39:18.952 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [ 2024-06-27T21:39:18.952 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:18.952 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:17 2024", 2024-06-27T21:39:18.952 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.532, 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.583, 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.591 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.658, 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.750, 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.750 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.953 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:18.954 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.658, 2024-06-27T21:39:18.954 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.750, 2024-06-27T21:39:18.954 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.750 2024-06-27T21:39:18.954 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.954 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.311 2024-06-27T21:39:18.954 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.954 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:18.954 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:15 2024", 2024-06-27T21:39:18.954 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:18.954 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:18.954 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:18.955 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:18.955 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:18.955 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.538, 2024-06-27T21:39:18.955 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.571, 2024-06-27T21:39:18.955 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.524 2024-06-27T21:39:18.955 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.955 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:18.955 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.605, 2024-06-27T21:39:18.955 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 1.128, 2024-06-27T21:39:18.955 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 1.128 2024-06-27T21:39:18.956 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.956 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:18.956 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.605, 2024-06-27T21:39:18.956 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 1.128, 2024-06-27T21:39:18.956 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 1.128 2024-06-27T21:39:18.956 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.956 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.374 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:17 2024", 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.490, 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.540, 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.546 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.957 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:18.958 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.685, 2024-06-27T21:39:18.958 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.733, 2024-06-27T21:39:18.958 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.733 2024-06-27T21:39:18.958 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.958 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:18.958 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.685, 2024-06-27T21:39:18.958 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.733, 2024-06-27T21:39:18.958 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.733 2024-06-27T21:39:18.958 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.393 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:15 2024", 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.459, 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.537, 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.482 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.959 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:18.960 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.568, 2024-06-27T21:39:18.960 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.984, 2024-06-27T21:39:18.960 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.984 2024-06-27T21:39:18.960 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.960 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:18.960 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.568, 2024-06-27T21:39:18.960 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.984, 2024-06-27T21:39:18.960 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.984 2024-06-27T21:39:18.960 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:18.960 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.269 2024-06-27T21:39:18.960 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T21:39:18.961 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T21:39:18.961 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:39:18.966 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:18.967 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:39:19.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-06-27T21:39:19.004 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:19.005 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-06-27T21:39:19.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:39:19.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:39:19.038 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-27T21:39:19.038 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-27T21:39:19.038 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-27T21:39:19.039 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:39:19.039 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:39:19.039 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:39:19.039 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-mgr.x.asok 2024-06-27T21:39:19.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:39:19.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-mgr.x.asok dump_osd_network 0 2024-06-27T21:39:19.145 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:39:19.145 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 0, 2024-06-27T21:39:19.146 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [ 2024-06-27T21:39:19.146 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.146 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:15 2024", 2024-06-27T21:39:19.146 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.146 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:39:19.146 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:19.146 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:19.146 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.146 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.618, 2024-06-27T21:39:19.146 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.595, 2024-06-27T21:39:19.146 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.607 2024-06-27T21:39:19.147 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.147 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.147 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.505, 2024-06-27T21:39:19.147 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.496, 2024-06-27T21:39:19.147 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.496 2024-06-27T21:39:19.147 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.147 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.147 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.707, 2024-06-27T21:39:19.148 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.775, 2024-06-27T21:39:19.148 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.775 2024-06-27T21:39:19.148 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.148 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.505 2024-06-27T21:39:19.148 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.148 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.148 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:11 2024", 2024-06-27T21:39:19.148 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.148 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:39:19.149 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:19.149 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:19.149 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.149 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.611, 2024-06-27T21:39:19.149 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.602, 2024-06-27T21:39:19.149 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.581 2024-06-27T21:39:19.149 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.149 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.150 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.570, 2024-06-27T21:39:19.150 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.445, 2024-06-27T21:39:19.150 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.445 2024-06-27T21:39:19.150 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.150 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.150 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.704, 2024-06-27T21:39:19.150 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.762, 2024-06-27T21:39:19.150 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.762 2024-06-27T21:39:19.151 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.151 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.614 2024-06-27T21:39:19.151 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.151 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.151 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:12 2024", 2024-06-27T21:39:19.151 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.151 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:19.151 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:19.151 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:19.152 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.152 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.563, 2024-06-27T21:39:19.152 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.595, 2024-06-27T21:39:19.152 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.595 2024-06-27T21:39:19.152 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.152 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.152 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.471, 2024-06-27T21:39:19.152 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.433, 2024-06-27T21:39:19.153 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.433 2024-06-27T21:39:19.153 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.153 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.153 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.645, 2024-06-27T21:39:19.153 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.750, 2024-06-27T21:39:19.153 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.750 2024-06-27T21:39:19.153 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.153 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.425 2024-06-27T21:39:19.154 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.154 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.154 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:14 2024", 2024-06-27T21:39:19.154 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.154 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:39:19.154 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:39:19.154 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:19.154 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.155 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.572, 2024-06-27T21:39:19.155 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.594, 2024-06-27T21:39:19.155 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.546 2024-06-27T21:39:19.155 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.155 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.155 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.483, 2024-06-27T21:39:19.155 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.361, 2024-06-27T21:39:19.155 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.361 2024-06-27T21:39:19.155 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.156 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.156 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.658, 2024-06-27T21:39:19.156 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.713, 2024-06-27T21:39:19.156 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.713 2024-06-27T21:39:19.156 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.156 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.526 2024-06-27T21:39:19.156 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.156 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.157 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:15 2024", 2024-06-27T21:39:19.157 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.157 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:39:19.157 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:39:19.157 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:19.157 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.157 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.537, 2024-06-27T21:39:19.157 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.560, 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.592 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.377, 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.377, 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.377 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.610, 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.667, 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.689 2024-06-27T21:39:19.158 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.377 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:11 2024", 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.494, 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.559, 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.585 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.159 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.438, 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.415, 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.415 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.540, 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.674, 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.674 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.483 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:15 2024", 2024-06-27T21:39:19.160 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.538, 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.571, 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.524 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.474, 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.464, 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.348 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.161 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.605, 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 1.128, 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 1.128 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.536 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:15 2024", 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.162 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.506, 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.559, 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.544 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.406, 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.406, 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.327 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.617, 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.672, 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.674 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.163 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.406 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:12 2024", 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.517, 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.552, 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.550 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.164 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.461, 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.398, 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.398 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.603, 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.733, 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.733 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.301 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:15 2024", 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:19.165 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.459, 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.537, 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.482 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.398, 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.368, 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.280 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.568, 2024-06-27T21:39:19.166 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.984, 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.984 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.398 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:15 2024", 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.513, 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.517, 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.491 2024-06-27T21:39:19.167 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.459, 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.425, 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.425 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.621, 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.710, 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.710 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.459 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:19.168 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:14 2024", 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.491, 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.508, 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.488 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.456, 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.436, 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.340 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.169 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:19.170 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.541, 2024-06-27T21:39:19.170 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.614, 2024-06-27T21:39:19.170 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.614 2024-06-27T21:39:19.170 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:19.170 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.456 2024-06-27T21:39:19.170 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T21:39:19.170 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T21:39:19.170 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:39:19.171 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:19.171 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:39:19.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-06-27T21:39:19.199 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:19.200 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-06-27T21:39:19.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:39:19.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-06-27T21:39:39.243 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-06-27T21:39:39.243 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:39:39.244 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:39:39.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T21:39:39.695 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T21:39:39.695 INFO:tasks.workunit.client.0.smithi042.stderr:2' 2024-06-27T21:39:39.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:39:39.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:39:39.695 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:39:39.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836513 2024-06-27T21:39:39.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836513 2024-06-27T21:39:39.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513' 2024-06-27T21:39:39.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:39:39.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T21:39:40.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-06-27T21:39:40.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-06-27T21:39:40.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-42949672985' 2024-06-27T21:39:40.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:39:40.009 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T21:39:40.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509458 2024-06-27T21:39:40.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509458 2024-06-27T21:39:40.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-42949672985 2-64424509458' 2024-06-27T21:39:40.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:39:40.166 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836513 2024-06-27T21:39:40.167 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:39:40.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:39:40.168 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836513 2024-06-27T21:39:40.168 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:39:40.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836513 2024-06-27T21:39:40.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-06-27T21:39:40.171 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836513 2024-06-27T21:39:40.171 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:39:40.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836513 -lt 21474836513 2024-06-27T21:39:40.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:39:40.604 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-06-27T21:39:40.604 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:39:40.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T21:39:40.607 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-06-27T21:39:40.607 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:39:40.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-06-27T21:39:40.608 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672985 2024-06-27T21:39:40.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-06-27T21:39:40.609 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T21:39:41.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672986 -lt 42949672985 2024-06-27T21:39:41.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:39:41.044 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509458 2024-06-27T21:39:41.045 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:39:41.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T21:39:41.047 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-06-27T21:39:41.047 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:39:41.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509458 2024-06-27T21:39:41.049 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509458 2024-06-27T21:39:41.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-06-27T21:39:41.050 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T21:39:41.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509458 2024-06-27T21:39:41.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:39:41.470 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-06-27T21:39:41.471 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:39:41.471 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:39:41.471 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:39:41.471 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:39:41.471 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:39:41.472 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-osd.0.asok 2024-06-27T21:39:41.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:39:41.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-osd.0.asok dump_osd_network 2024-06-27T21:39:41.583 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:39:41.583 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 1000, 2024-06-27T21:39:41.583 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [] 2024-06-27T21:39:41.584 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:39:41.597 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:41.598 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:39:41.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:39:41.628 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:41.629 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-06-27T21:39:41.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-06-27T21:39:41.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:39:41.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-27T21:39:41.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-27T21:39:41.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-27T21:39:41.670 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:39:41.670 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:39:41.670 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:39:41.670 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-mgr.x.asok 2024-06-27T21:39:41.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:39:41.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-mgr.x.asok dump_osd_network 2024-06-27T21:39:41.768 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:39:41.769 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 1000, 2024-06-27T21:39:41.769 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [] 2024-06-27T21:39:41.769 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:39:41.782 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:41.783 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:39:41.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:39:41.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:41.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-06-27T21:39:41.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-06-27T21:39:41.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:39:41.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-06-27T21:39:41.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:39:41.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:39:41.852 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:39:41.852 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:39:41.853 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:39:41.853 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-osd.0.asok 2024-06-27T21:39:41.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:39:41.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-osd.0.asok dump_osd_network 0 2024-06-27T21:39:41.960 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:39:41.961 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 0, 2024-06-27T21:39:41.961 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [ 2024-06-27T21:39:41.961 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:41.961 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:37 2024", 2024-06-27T21:39:41.961 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:41.961 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:41.961 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:41.962 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:41.962 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:41.962 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.580, 2024-06-27T21:39:41.962 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.562, 2024-06-27T21:39:41.962 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.584 2024-06-27T21:39:41.962 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.962 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:41.962 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.675, 2024-06-27T21:39:41.962 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.690, 2024-06-27T21:39:41.962 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.750 2024-06-27T21:39:41.962 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.963 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:41.963 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.675, 2024-06-27T21:39:41.963 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.690, 2024-06-27T21:39:41.963 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.750 2024-06-27T21:39:41.963 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.963 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.277 2024-06-27T21:39:41.963 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.963 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:41.963 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:37 2024", 2024-06-27T21:39:41.963 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.503, 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.533, 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.545 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.629, 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.685, 2024-06-27T21:39:41.964 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.733 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.629, 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.685, 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.733 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.250 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:40 2024", 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:41.965 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:41.966 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:41.966 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:41.966 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.521, 2024-06-27T21:39:41.966 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.539, 2024-06-27T21:39:41.966 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.543 2024-06-27T21:39:41.966 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.966 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:41.966 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.585, 2024-06-27T21:39:41.966 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.745, 2024-06-27T21:39:41.966 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 1.128 2024-06-27T21:39:41.967 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.967 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:41.967 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.585, 2024-06-27T21:39:41.967 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.745, 2024-06-27T21:39:41.967 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 1.128 2024-06-27T21:39:41.967 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.967 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.300 2024-06-27T21:39:41.967 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.967 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:41.967 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:40 2024", 2024-06-27T21:39:41.967 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.523, 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.504, 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.510 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.598, 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.670, 2024-06-27T21:39:41.968 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.984 2024-06-27T21:39:41.969 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.969 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:41.969 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.598, 2024-06-27T21:39:41.969 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.670, 2024-06-27T21:39:41.969 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.984 2024-06-27T21:39:41.969 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:41.969 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.334 2024-06-27T21:39:41.969 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T21:39:41.969 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T21:39:41.969 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:39:41.975 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:41.975 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:39:42.011 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-06-27T21:39:42.013 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:42.013 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-06-27T21:39:42.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:39:42.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:39:42.045 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-27T21:39:42.045 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-27T21:39:42.045 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-27T21:39:42.045 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:39:42.045 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:39:42.045 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:39:42.045 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-mgr.x.asok 2024-06-27T21:39:42.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:39:42.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-mgr.x.asok dump_osd_network 0 2024-06-27T21:39:42.148 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:39:42.148 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 0, 2024-06-27T21:39:42.148 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [ 2024-06-27T21:39:42.148 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.149 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:35 2024", 2024-06-27T21:39:42.149 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.149 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:39:42.149 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:42.149 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:42.149 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.149 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.550, 2024-06-27T21:39:42.149 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.580, 2024-06-27T21:39:42.149 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.595 2024-06-27T21:39:42.149 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.149 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.150 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.498, 2024-06-27T21:39:42.150 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.448, 2024-06-27T21:39:42.150 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.448 2024-06-27T21:39:42.150 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.150 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.150 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.652, 2024-06-27T21:39:42.150 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.707, 2024-06-27T21:39:42.150 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.775 2024-06-27T21:39:42.150 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.150 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.445 2024-06-27T21:39:42.150 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:36 2024", 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.589, 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.581, 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.585 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.151 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.152 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.548, 2024-06-27T21:39:42.152 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.345, 2024-06-27T21:39:42.152 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.345 2024-06-27T21:39:42.152 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.152 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.152 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.648, 2024-06-27T21:39:42.152 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.696, 2024-06-27T21:39:42.152 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.762 2024-06-27T21:39:42.152 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.152 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.468 2024-06-27T21:39:42.152 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.153 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.153 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:37 2024", 2024-06-27T21:39:42.153 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.153 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:42.153 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:42.153 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:42.153 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.153 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.580, 2024-06-27T21:39:42.153 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.562, 2024-06-27T21:39:42.153 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.584 2024-06-27T21:39:42.153 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.154 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.154 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.526, 2024-06-27T21:39:42.154 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.311, 2024-06-27T21:39:42.154 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.311 2024-06-27T21:39:42.154 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.154 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.154 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.675, 2024-06-27T21:39:42.154 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.690, 2024-06-27T21:39:42.154 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.750 2024-06-27T21:39:42.154 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.154 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.461 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:36 2024", 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.563, 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.572, 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.577 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.155 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.156 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.471, 2024-06-27T21:39:42.156 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.297, 2024-06-27T21:39:42.156 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.297 2024-06-27T21:39:42.156 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.156 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.156 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.639, 2024-06-27T21:39:42.156 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.663, 2024-06-27T21:39:42.156 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.674 2024-06-27T21:39:42.156 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.156 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.602 2024-06-27T21:39:42.156 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.157 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.157 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:35 2024", 2024-06-27T21:39:42.157 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.157 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:39:42.157 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:39:42.157 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:42.157 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.157 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.543, 2024-06-27T21:39:42.157 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.540, 2024-06-27T21:39:42.157 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.574 2024-06-27T21:39:42.157 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.158 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.158 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.451, 2024-06-27T21:39:42.158 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.377, 2024-06-27T21:39:42.158 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.377 2024-06-27T21:39:42.158 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.158 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.158 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.627, 2024-06-27T21:39:42.158 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.633, 2024-06-27T21:39:42.158 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.689 2024-06-27T21:39:42.158 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.158 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.364 2024-06-27T21:39:42.159 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.159 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.159 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:35 2024", 2024-06-27T21:39:42.159 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.159 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:39:42.159 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:42.159 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:42.159 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.159 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.533, 2024-06-27T21:39:42.159 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.533, 2024-06-27T21:39:42.159 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.545 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.488, 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.399, 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.327 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.636, 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.636, 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.674 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.160 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.393 2024-06-27T21:39:42.161 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.161 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.161 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:37 2024", 2024-06-27T21:39:42.161 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.161 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:42.161 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:39:42.161 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:42.161 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.161 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.503, 2024-06-27T21:39:42.161 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.533, 2024-06-27T21:39:42.162 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.545 2024-06-27T21:39:42.162 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.162 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.162 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.429, 2024-06-27T21:39:42.162 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.301, 2024-06-27T21:39:42.162 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.301 2024-06-27T21:39:42.162 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.162 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.162 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.629, 2024-06-27T21:39:42.162 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.685, 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.733 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.575 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:39 2024", 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.163 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.501, 2024-06-27T21:39:42.164 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.514, 2024-06-27T21:39:42.164 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.543 2024-06-27T21:39:42.164 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.164 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.164 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.402, 2024-06-27T21:39:42.164 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.271, 2024-06-27T21:39:42.164 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.271 2024-06-27T21:39:42.164 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.164 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.164 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.572, 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.645, 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.713 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.402 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:35 2024", 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:39:42.165 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.166 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.535, 2024-06-27T21:39:42.166 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.542, 2024-06-27T21:39:42.166 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.540 2024-06-27T21:39:42.166 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.166 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.166 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.422, 2024-06-27T21:39:42.166 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.282, 2024-06-27T21:39:42.166 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.282 2024-06-27T21:39:42.166 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.166 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.166 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.640, 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.745, 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 1.128 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.497 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:35 2024", 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:42.167 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.523, 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.507, 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.498 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.428, 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.427, 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.425 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.621, 2024-06-27T21:39:42.168 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.621, 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.710 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.339 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:35 2024", 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.169 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.522, 2024-06-27T21:39:42.170 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.491, 2024-06-27T21:39:42.170 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.502 2024-06-27T21:39:42.170 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.170 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.170 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.479, 2024-06-27T21:39:42.170 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.264, 2024-06-27T21:39:42.170 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.264 2024-06-27T21:39:42.170 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.170 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.170 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.616, 2024-06-27T21:39:42.170 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.670, 2024-06-27T21:39:42.171 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.984 2024-06-27T21:39:42.171 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.171 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.535 2024-06-27T21:39:42.171 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.171 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:39:42.171 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:39:39 2024", 2024-06-27T21:39:42.171 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:39:42.171 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:39:42.171 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:39:42.171 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.517, 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.499, 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.496 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.433, 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.285, 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.285 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:39:42.172 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.609, 2024-06-27T21:39:42.173 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.609, 2024-06-27T21:39:42.173 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.614 2024-06-27T21:39:42.173 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:39:42.173 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.545 2024-06-27T21:39:42.173 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T21:39:42.173 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T21:39:42.173 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:39:42.174 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:42.174 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:39:42.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-06-27T21:39:42.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:39:42.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-06-27T21:39:42.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:39:42.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-06-27T21:40:32.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-06-27T21:40:32.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:40:32.241 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:40:32.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T21:40:32.689 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T21:40:32.689 INFO:tasks.workunit.client.0.smithi042.stderr:2' 2024-06-27T21:40:32.689 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:40:32.689 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:40:32.689 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:40:32.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836525 2024-06-27T21:40:32.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836525 2024-06-27T21:40:32.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836525' 2024-06-27T21:40:32.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:40:32.862 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T21:40:33.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672998 2024-06-27T21:40:33.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672998 2024-06-27T21:40:33.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836525 1-42949672998' 2024-06-27T21:40:33.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:40:33.019 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T21:40:33.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509471 2024-06-27T21:40:33.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509471 2024-06-27T21:40:33.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836525 1-42949672998 2-64424509471' 2024-06-27T21:40:33.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:40:33.178 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836525 2024-06-27T21:40:33.178 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:40:33.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:40:33.180 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836525 2024-06-27T21:40:33.181 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:40:33.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836525 2024-06-27T21:40:33.182 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836525 2024-06-27T21:40:33.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-06-27T21:40:33.182 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:40:33.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836523 -lt 21474836525 2024-06-27T21:40:33.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T21:40:34.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T21:40:34.659 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:40:35.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836526 -lt 21474836525 2024-06-27T21:40:35.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:40:35.106 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672998 2024-06-27T21:40:35.106 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:40:35.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T21:40:35.108 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672998 2024-06-27T21:40:35.108 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:40:35.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672998 2024-06-27T21:40:35.110 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672998 2024-06-27T21:40:35.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-06-27T21:40:35.110 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T21:40:35.536 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672998 -lt 42949672998 2024-06-27T21:40:35.536 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:40:35.537 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509471 2024-06-27T21:40:35.537 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:40:35.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T21:40:35.539 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509471 2024-06-27T21:40:35.539 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:40:35.540 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509471 2024-06-27T21:40:35.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509471 2024-06-27T21:40:35.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-06-27T21:40:35.541 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T21:40:35.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509471 -lt 64424509471 2024-06-27T21:40:35.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:40:35.976 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-06-27T21:40:35.976 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:40:35.976 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:40:35.977 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:40:35.977 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:35.977 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:35.978 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-osd.0.asok 2024-06-27T21:40:35.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:40:35.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-osd.0.asok dump_osd_network 2024-06-27T21:40:36.086 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:40:36.087 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 1000, 2024-06-27T21:40:36.087 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [] 2024-06-27T21:40:36.087 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:40:36.101 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:40:36.101 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:40:36.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:40:36.136 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:40:36.136 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-06-27T21:40:36.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-06-27T21:40:36.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:40:36.170 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-27T21:40:36.170 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-27T21:40:36.170 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-27T21:40:36.171 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:40:36.171 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:36.171 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:36.171 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-mgr.x.asok 2024-06-27T21:40:36.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:40:36.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-mgr.x.asok dump_osd_network 2024-06-27T21:40:36.271 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:40:36.271 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 1000, 2024-06-27T21:40:36.271 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [] 2024-06-27T21:40:36.271 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:40:36.286 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:40:36.286 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:40:36.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:40:36.323 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:40:36.323 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-06-27T21:40:36.356 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-06-27T21:40:36.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:40:36.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-06-27T21:40:36.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:40:36.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:40:36.358 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:40:36.358 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:36.358 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:36.358 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-osd.0.asok 2024-06-27T21:40:36.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:40:36.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-osd.0.asok dump_osd_network 0 2024-06-27T21:40:36.465 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:40:36.465 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 0, 2024-06-27T21:40:36.465 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [ 2024-06-27T21:40:36.465 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.465 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:32 2024", 2024-06-27T21:40:36.465 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.622, 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.572, 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.571 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.670, 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.761, 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.761 2024-06-27T21:40:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.670, 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.761, 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.761 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.252 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:35 2024", 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:40:36.467 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.495, 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.535, 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.543 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.600, 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.617, 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.667 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.600, 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.617, 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.667 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.320 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:32 2024", 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:40:36.469 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.522, 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.535, 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.534 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.557, 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.681, 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.681 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.557, 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.681, 2024-06-27T21:40:36.470 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.681 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.280 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:35 2024", 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.450, 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.514, 2024-06-27T21:40:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.516 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.676, 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.676, 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.683 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.676, 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.676, 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.683 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.337 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T21:40:36.472 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T21:40:36.473 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:40:36.481 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:40:36.481 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:40:36.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-06-27T21:40:36.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:40:36.518 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-06-27T21:40:36.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:40:36.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:40:36.561 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-27T21:40:36.561 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-27T21:40:36.561 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-27T21:40:36.561 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:40:36.561 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:36.561 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:36.561 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-mgr.x.asok 2024-06-27T21:40:36.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:40:36.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-mgr.x.asok dump_osd_network 0 2024-06-27T21:40:36.668 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:40:36.668 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 0, 2024-06-27T21:40:36.668 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [ 2024-06-27T21:40:36.668 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.668 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:32 2024", 2024-06-27T21:40:36.668 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.668 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.622, 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.572, 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.571 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.580, 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.429, 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.277 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.669 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.670, 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.761, 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.761 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.424 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:31 2024", 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.609, 2024-06-27T21:40:36.670 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.600, 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.593 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.534, 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.502, 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.345 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.669, 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.711, 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.713 2024-06-27T21:40:36.671 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.617 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:35 2024", 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.585, 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.584, 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.577 2024-06-27T21:40:36.672 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.463, 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.435, 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.435 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.700, 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.700, 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.737 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.618 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.673 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:35 2024", 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.518, 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.574, 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.550 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.382, 2024-06-27T21:40:36.674 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.382, 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.359 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.640, 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.694, 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.716 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.577 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:31 2024", 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:40:36.675 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.511, 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.561, 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.572 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.450, 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.443, 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.297 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.651, 2024-06-27T21:40:36.676 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.686, 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.701 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.653 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:30 2024", 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.565, 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.544, 2024-06-27T21:40:36.677 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.543 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.540, 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.454, 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.282 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.589, 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.617, 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.745 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.315 2024-06-27T21:40:36.678 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:34 2024", 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.525, 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.560, 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.548 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.420, 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.420, 2024-06-27T21:40:36.679 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.402 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.591, 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.680, 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.727 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.489 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:35 2024", 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:40:36.680 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "front", 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.536, 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.551, 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.548 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.423, 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.423, 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.337 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.624, 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.703, 2024-06-27T21:40:36.681 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.704 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.486 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:32 2024", 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 2, 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.522, 2024-06-27T21:40:36.682 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.535, 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.534 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.450, 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.444, 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.250 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.557, 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.681, 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.681 2024-06-27T21:40:36.683 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.383 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:35 2024", 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 2, 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.506, 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.520, 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.530 2024-06-27T21:40:36.684 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.685 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.685 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.393, 2024-06-27T21:40:36.685 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.393, 2024-06-27T21:40:36.685 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.339 2024-06-27T21:40:36.685 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.685 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.685 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.585, 2024-06-27T21:40:36.685 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.658, 2024-06-27T21:40:36.685 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.663 2024-06-27T21:40:36.685 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.685 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.393 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:30 2024", 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 0, 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 1, 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.525, 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.520, 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.514 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.686 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.489, 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.408, 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.264 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.626, 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.664, 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.683 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.286 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T21:40:36.687 INFO:tasks.workunit.client.0.smithi042.stdout: "last update": "Thu Jun 27 21:40:34 2024", 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "stale": false, 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "from osd": 1, 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "to osd": 0, 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "interface": "back", 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "average": { 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.518, 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.521, 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.514 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "min": { 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.407, 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.407, 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.397 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.688 INFO:tasks.workunit.client.0.smithi042.stdout: "max": { 2024-06-27T21:40:36.689 INFO:tasks.workunit.client.0.smithi042.stdout: "1min": 0.591, 2024-06-27T21:40:36.689 INFO:tasks.workunit.client.0.smithi042.stdout: "5min": 0.676, 2024-06-27T21:40:36.689 INFO:tasks.workunit.client.0.smithi042.stdout: "15min": 0.676 2024-06-27T21:40:36.689 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T21:40:36.689 INFO:tasks.workunit.client.0.smithi042.stdout: "last": 0.571 2024-06-27T21:40:36.689 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T21:40:36.689 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T21:40:36.689 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:40:36.690 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:40:36.690 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-06-27T21:40:36.720 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-06-27T21:40:36.722 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:40:36.722 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-06-27T21:40:36.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-06-27T21:40:36.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:40:36.761 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-06-27T21:40:36.761 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-06-27T21:40:36.762 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-06-27T21:40:36.762 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:40:36.763 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:36.763 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:36.763 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-mgr.x.asok 2024-06-27T21:40:36.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:40:36.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-mgr.x.asok dump_osd_network 99 2024-06-27T21:40:36.869 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:40:36.869 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 99, 2024-06-27T21:40:36.869 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [] 2024-06-27T21:40:36.869 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:40:36.885 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:40:36.885 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-06-27T21:40:36.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-06-27T21:40:36.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-06-27T21:40:36.924 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-06-27T21:40:36.924 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:40:36.924 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:40:36.924 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:40:36.924 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:36.924 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:36.924 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-osd.0.asok 2024-06-27T21:40:36.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-06-27T21:40:36.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.34572/ceph-osd.0.asok dump_osd_network 98 2024-06-27T21:40:37.030 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T21:40:37.030 INFO:tasks.workunit.client.0.smithi042.stdout: "threshold": 98, 2024-06-27T21:40:37.030 INFO:tasks.workunit.client.0.smithi042.stdout: "entries": [] 2024-06-27T21:40:37.031 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T21:40:37.046 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-06-27T21:40:37.046 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-06-27T21:40:37.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-06-27T21:40:37.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-06-27T21:40:37.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-06-27T21:40:37.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-06-27T21:40:37.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:40:37.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-06-27T21:40:37.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:40:37.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:40:37.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:40:37.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:40:37.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:40:37.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:40:37.399 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:40:37.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:40:37.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:40:37.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:40:37.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:40:37.403 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:40:37.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:40:37.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:40:37.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:40:37.406 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:40:37.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:40:37.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:40:37.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-06-27T21:40:37.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:40:37.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:37.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:37.437 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34572 2024-06-27T21:40:37.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:40:37.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:40:37.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-06-27T21:40:37.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-06-27T21:40:37.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-06-27T21:40:37.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-06-27T21:40:37.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-06-27T21:40:37.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:40:37.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-06-27T21:40:37.440 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:40:37.440 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:40:37.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:40:37.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:40:37.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:40:37.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:40:37.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:40:37.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:40:37.445 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:40:37.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:40:37.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:40:37.446 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:40:37.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:40:37.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:40:37.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:40:37.448 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:40:37.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:40:37.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:40:37.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-06-27T21:40:37.456 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:40:37.456 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:37.456 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:37.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34572 2024-06-27T21:40:37.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:40:37.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:40:37.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-06-27T21:40:37.456 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:40:37.456 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:37.456 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:37.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.34572 2024-06-27T21:40:37.458 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:40:37.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:40:37.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:40:37.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-06-27T21:40:37.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-06-27T21:40:37.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-06-27T21:40:37.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-06-27T21:40:37.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-06-27T21:40:37.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-06-27T21:40:37.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-06-27T21:40:37.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-06-27T21:40:37.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:40:37.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:40:37.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:40:37.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-06-27T21:40:37.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-06-27T21:40:37.911 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:40:37.911 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:40:37.911 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:40:37.911 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:40:37.911 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:37.911 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:37.912 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:40:37.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:40:38.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:40:38.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:40:38.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:40:38.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:40:38.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:40:38.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:40:38.140 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:40:38.140 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:40:38.140 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:40:38.141 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:40:38.141 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:38.141 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:38.141 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-mon.a.asok 2024-06-27T21:40:38.141 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:40:38.141 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34572/ceph-mon.a.asok config get fsid 2024-06-27T21:40:38.260 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:40:38.260 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:40:38.260 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:40:38.260 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:40:38.261 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:40:38.262 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:40:38.262 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:40:38.262 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:40:38.262 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:40:38.262 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:38.262 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:38.263 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.34572/ceph-mon.a.asok 2024-06-27T21:40:38.263 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:40:38.263 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.34572/ceph-mon.a.asok config get mon_host 2024-06-27T21:40:38.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-06-27T21:40:38.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-06-27T21:40:38.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:40:38.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:40:38.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:40:38.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-06-27T21:40:38.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:40:39.306 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:40:39.306 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:40:39.306 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:40:39.306 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:40:39.306 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:39.306 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:39.306 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:40:39.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:40:39.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:40:39.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-06-27T21:40:39.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-27T21:40:39.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:40:39.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:40:39.389 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:40:39.390 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:40:39.390 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:40:39.390 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:40:39.390 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:40:39.390 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:40:39.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:40:39.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:40:39.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef18df50-e89c-4edd-9f28-07cee63b70c8 2024-06-27T21:40:39.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ef18df50-e89c-4edd-9f28-07cee63b70c8' 2024-06-27T21:40:39.393 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 ef18df50-e89c-4edd-9f28-07cee63b70c8 2024-06-27T21:40:39.393 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:40:39.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBX3H1mQdtiGBAAhJNEflS5RM/2AgGpu4QOdg== 2024-06-27T21:40:39.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBX3H1mQdtiGBAAhJNEflS5RM/2AgGpu4QOdg=="}' 2024-06-27T21:40:39.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef18df50-e89c-4edd-9f28-07cee63b70c8 -i td/network-ping/0/new.json 2024-06-27T21:40:39.734 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:40:39.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-06-27T21:40:39.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBX3H1mQdtiGBAAhJNEflS5RM/2AgGpu4QOdg== --osd-uuid ef18df50-e89c-4edd-9f28-07cee63b70c8 2024-06-27T21:40:39.788 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:39.781+0000 7f6ff992f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:40:39.811 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:39.805+0000 7f6ff992f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:40:39.812 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:39.805+0000 7f6ff992f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:40:39.812 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:39.805+0000 7f6ff992f7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-27T21:40:39.812 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:39.805+0000 7f6ff992f7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-27T21:40:39.812 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:39.805+0000 7f6ff992f7c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-06-27T21:40:39.850 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:39.845+0000 7f6ff992f7c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-06-27T21:40:46.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-06-27T21:40:46.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:40:46.423 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:40:46.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:40:46.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:40:47.016 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:40:47.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:40:47.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:40:47.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:40:47.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:40:47.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:40:47.051 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:47.044+0000 7f029a5e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:40:47.054 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:47.048+0000 7f029a5e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:40:47.055 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:47.048+0000 7f029a5e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:40:47.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:40:47.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:40:47.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:40:47.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:40:47.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:40:47.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:40:47.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:40:47.492 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:40:47.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:40:47.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:40:47.623 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:47.616+0000 7f029a5e27c0 -1 Falling back to public interface 2024-06-27T21:40:47.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:40:48.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:40:48.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:40:48.957 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:40:48.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:40:48.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:40:48.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:40:49.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:40:50.239 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:40:50.232+0000 7f029a5e27c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:40:50.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:40:50.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:40:50.394 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:40:50.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:40:50.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:40:50.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:40:50.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:40:51.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:40:51.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:40:51.946 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:40:51.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:40:51.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:40:51.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:40:52.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:40:53.426 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:40:53.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:40:53.429 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:40:53.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:40:53.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:40:53.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:40:53.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:40:54.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:40:54.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:40:54.889 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:40:54.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:40:54.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:40:54.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:40:55.427 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:40:56.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:40:56.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:40:56.429 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:40:56.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:40:56.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:40:56.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:40:56.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:40:57.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:40:57.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:40:57.898 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:40:57.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:40:57.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:40:57.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:40:58.437 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:40:59.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:40:59.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:40:59.439 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:40:59.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:40:59.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:40:59.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:40:59.884 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:00.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:00.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:00.886 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:41:00.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:41:00.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:00.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:41:01.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:02.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:02.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:02.410 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:41:02.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:41:02.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:02.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:41:03.075 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:04.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:04.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:04.077 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:41:04.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:41:04.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:04.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:41:04.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:05.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:05.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:05.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:41:05.674 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:41:05.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:05.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:41:06.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:07.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:07.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:07.410 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:41:07.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:41:07.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:07.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:41:08.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:09.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:09.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:09.204 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:41:09.204 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:41:09.204 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:09.204 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:41:10.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:11.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:11.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:11.300 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:41:11.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:41:11.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:11.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:41:12.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:13.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:13.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:13.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:41:13.423 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:41:13.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:13.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:41:14.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:15.052 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:15.052 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:15.053 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:41:15.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:41:15.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:15.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:41:15.482 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2961416859,v1:127.0.0.1:6803/2961416859] [v2:127.0.0.1:6804/2961416859,v1:127.0.0.1:6805/2961416859] exists,up ef18df50-e89c-4edd-9f28-07cee63b70c8 2024-06-27T21:41:15.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:41:15.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:41:15.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:41:15.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-06-27T21:41:15.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-27T21:41:15.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:41:15.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T21:41:15.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:41:15.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-06-27T21:41:15.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:41:15.486 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:41:15.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:41:15.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:41:15.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:41:15.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-06-27T21:41:15.488 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:41:15.488 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 2fa2a112-4c62-4e57-8897-58c7148963fa 2024-06-27T21:41:15.489 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2fa2a112-4c62-4e57-8897-58c7148963fa 2024-06-27T21:41:15.489 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2fa2a112-4c62-4e57-8897-58c7148963fa' 2024-06-27T21:41:15.489 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:41:15.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB73H1mYLA5HhAAc/kIYMdHmdCGAZgMoS4QhA== 2024-06-27T21:41:15.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB73H1mYLA5HhAAc/kIYMdHmdCGAZgMoS4QhA=="}' 2024-06-27T21:41:15.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2fa2a112-4c62-4e57-8897-58c7148963fa -i td/network-ping/1/new.json 2024-06-27T21:41:16.044 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:41:16.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-06-27T21:41:16.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB73H1mYLA5HhAAc/kIYMdHmdCGAZgMoS4QhA== --osd-uuid 2fa2a112-4c62-4e57-8897-58c7148963fa 2024-06-27T21:41:16.100 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:16.092+0000 7fee0647e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:16.102 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:16.096+0000 7fee0647e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:16.104 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:16.096+0000 7fee0647e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:16.104 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:16.096+0000 7fee0647e7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-27T21:41:16.104 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:16.096+0000 7fee0647e7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-27T21:41:16.104 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:16.096+0000 7fee0647e7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-06-27T21:41:16.147 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:16.140+0000 7fee0647e7c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-06-27T21:41:22.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-06-27T21:41:22.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:41:22.344 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T21:41:22.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T21:41:22.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:41:22.923 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T21:41:22.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T21:41:22.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:41:22.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:41:22.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:41:22.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:41:22.958 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:22.952+0000 7f042e6cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:22.970 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:22.964+0000 7f042e6cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:22.972 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:22.964+0000 7f042e6cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:23.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T21:41:23.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:41:23.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T21:41:23.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:41:23.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:41:23.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:23.382 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:41:23.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:41:23.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:23.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:23.535 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:23.528+0000 7f042e6cb7c0 -1 Falling back to public interface 2024-06-27T21:41:23.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:24.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:24.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:24.860 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:41:24.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:41:24.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:24.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:25.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:26.087 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:26.080+0000 7f042e6cb7c0 -1 osd.1 0 log_to_monitors true 2024-06-27T21:41:26.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:26.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:26.318 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:41:26.318 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:41:26.318 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:26.318 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:26.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:27.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:27.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:27.936 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:41:27.937 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:41:27.937 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:27.937 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:28.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:29.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:29.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:29.375 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:41:29.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:41:29.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:29.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:29.984 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:30.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:30.987 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:41:30.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:30.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:41:30.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:30.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:31.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:32.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:32.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:32.460 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:41:32.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:41:32.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:32.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:32.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:33.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:33.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:33.939 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:41:33.939 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:41:33.939 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:33.939 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:34.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:35.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:35.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:35.422 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:41:35.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:41:35.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:35.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:35.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:36.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:36.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:36.923 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:41:36.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:41:36.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:36.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:37.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:38.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:38.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:38.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:41:38.457 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:41:38.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:38.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:38.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:39.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:39.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:39.972 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:41:39.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:41:39.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:39.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:40.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:41.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:41.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:41.635 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:41:41.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:41:41.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:41.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:42.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:43.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:43.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:43.365 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:41:43.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:41:43.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:43.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:44.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:45.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:45.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:45.094 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:41:45.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:41:45.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:45.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:45.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:46.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:46.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:46.893 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:41:46.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:41:46.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:46.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:47.770 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:48.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:48.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:48.772 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:41:48.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:41:48.773 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:48.773 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:49.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:41:50.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:41:50.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:50.544 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:41:50.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:41:50.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:50.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:41:51.052 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/304820926,v1:127.0.0.1:6811/304820926] [v2:127.0.0.1:6812/304820926,v1:127.0.0.1:6813/304820926] exists,up 2fa2a112-4c62-4e57-8897-58c7148963fa 2024-06-27T21:41:51.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:41:51.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:41:51.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:41:51.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-06-27T21:41:51.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-06-27T21:41:51.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:41:51.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:41:51.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:41:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:41:51.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:41:51.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:41:51.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:41:51.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:41:51.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-06-27T21:41:51.059 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:41:51.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=88c813b2-c356-44b3-bbdd-92e3aaf80612 2024-06-27T21:41:51.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 88c813b2-c356-44b3-bbdd-92e3aaf80612' 2024-06-27T21:41:51.060 INFO:tasks.workunit.client.0.smithi042.stdout:add osd2 88c813b2-c356-44b3-bbdd-92e3aaf80612 2024-06-27T21:41:51.060 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:41:51.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCf3H1mNHmmBBAAYBC+yWoEjeRO3P30toqHSQ== 2024-06-27T21:41:51.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCf3H1mNHmmBBAAYBC+yWoEjeRO3P30toqHSQ=="}' 2024-06-27T21:41:51.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 88c813b2-c356-44b3-bbdd-92e3aaf80612 -i td/network-ping/2/new.json 2024-06-27T21:41:51.632 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:41:51.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-06-27T21:41:51.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCf3H1mNHmmBBAAYBC+yWoEjeRO3P30toqHSQ== --osd-uuid 88c813b2-c356-44b3-bbdd-92e3aaf80612 2024-06-27T21:41:51.686 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:51.680+0000 7fe5679dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:51.690 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:51.688+0000 7fe5679dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:51.691 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:51.688+0000 7fe5679dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:51.691 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:51.688+0000 7fe5679dc7c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-27T21:41:51.692 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:51.688+0000 7fe5679dc7c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-27T21:41:51.692 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:51.688+0000 7fe5679dc7c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-06-27T21:41:51.735 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:51.732+0000 7fe5679dc7c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-06-27T21:41:58.426 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-06-27T21:41:58.426 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:41:58.429 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd2 key to auth repository 2024-06-27T21:41:58.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-27T21:41:58.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:41:59.010 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T21:41:59.011 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-27T21:41:59.011 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ffbf7189-804c-41bf-8391-8f9a6139e6ff --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.34572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:41:59.011 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:41:59.011 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:41:59.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:41:59.045 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:59.040+0000 7f1a0ee967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:59.060 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:59.056+0000 7f1a0ee967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:59.061 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:59.056+0000 7f1a0ee967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:41:59.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-27T21:41:59.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:41:59.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T21:41:59.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:41:59.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:41:59.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:41:59.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:41:59.492 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:41:59.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:41:59.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:41:59.627 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:41:59.624+0000 7f1a0ee967c0 -1 Falling back to public interface 2024-06-27T21:41:59.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:00.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:00.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:00.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:42:00.949 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:42:00.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:00.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:01.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:02.235 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:42:02.232+0000 7f1a0ee967c0 -1 osd.2 0 log_to_monitors true 2024-06-27T21:42:02.384 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:02.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:02.385 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:42:02.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:42:02.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:02.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:02.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:03.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:03.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:03.997 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:42:03.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:42:03.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:03.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:04.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:05.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:05.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:05.525 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:42:05.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:42:05.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:05.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:05.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:06.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:06.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:06.966 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:42:06.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:42:06.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:06.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:07.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:08.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:08.489 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:08.489 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:42:08.489 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:42:08.490 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:08.490 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:09.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:10.052 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:10.052 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:10.053 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:42:10.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:42:10.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:10.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:10.490 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:11.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:11.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:11.492 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:42:11.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:42:11.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:11.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:11.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:12.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:12.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:12.935 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:42:12.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:42:12.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:12.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:13.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:14.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:14.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:14.419 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:42:14.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:42:14.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:14.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:15.112 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:16.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:16.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:16.114 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:42:16.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:42:16.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:16.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:16.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:17.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:17.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:17.594 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:42:17.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:42:17.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:17.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:18.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:19.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:19.630 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:42:19.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:19.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:42:19.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:19.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:20.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:21.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:21.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:21.293 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:42:21.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:42:21.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:21.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:22.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:23.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:23.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:23.414 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:42:23.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:42:23.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:23.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:24.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:25.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:25.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:25.350 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:42:25.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:42:25.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:25.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:26.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:27.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:27.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:27.169 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:42:27.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:42:27.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:27.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:28.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:42:29.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:42:29.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:42:29.132 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T21:42:29.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T21:42:29.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:42:29.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:42:29.606 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3809201164,v1:127.0.0.1:6819/3809201164] [v2:127.0.0.1:6820/3809201164,v1:127.0.0.1:6821/3809201164] exists,up 88c813b2-c356-44b3-bbdd-92e3aaf80612 2024-06-27T21:42:29.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:42:29.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:42:29.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:42:29.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-06-27T21:42:34.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-06-27T21:42:35.162 INFO:tasks.workunit.client.0.smithi042.stderr:bucket 'dc1' already exists 2024-06-27T21:42:35.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-06-27T21:42:35.781 INFO:tasks.workunit.client.0.smithi042.stderr:bucket 'dc2' already exists 2024-06-27T21:42:35.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-06-27T21:42:36.402 INFO:tasks.workunit.client.0.smithi042.stderr:bucket 'dc3' already exists 2024-06-27T21:42:36.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-06-27T21:42:36.975 INFO:tasks.workunit.client.0.smithi042.stderr:bucket 'rack1' already exists 2024-06-27T21:42:36.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-06-27T21:42:37.564 INFO:tasks.workunit.client.0.smithi042.stderr:bucket 'rack2' already exists 2024-06-27T21:42:37.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-06-27T21:42:38.178 INFO:tasks.workunit.client.0.smithi042.stderr:bucket 'rack3' already exists 2024-06-27T21:42:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-06-27T21:42:38.801 INFO:tasks.workunit.client.0.smithi042.stderr:bucket 'host1' already exists 2024-06-27T21:42:38.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-06-27T21:42:39.373 INFO:tasks.workunit.client.0.smithi042.stderr:bucket 'host2' already exists 2024-06-27T21:42:39.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-06-27T21:42:39.962 INFO:tasks.workunit.client.0.smithi042.stderr:bucket 'host3' already exists 2024-06-27T21:42:39.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-06-27T21:42:40.620 INFO:tasks.workunit.client.0.smithi042.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-06-27T21:42:40.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-06-27T21:42:41.183 INFO:tasks.workunit.client.0.smithi042.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-06-27T21:42:41.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-06-27T21:42:41.785 INFO:tasks.workunit.client.0.smithi042.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-06-27T21:42:41.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-06-27T21:42:42.351 INFO:tasks.workunit.client.0.smithi042.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-06-27T21:42:42.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-06-27T21:42:42.947 INFO:tasks.workunit.client.0.smithi042.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-06-27T21:42:42.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-06-27T21:42:43.510 INFO:tasks.workunit.client.0.smithi042.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-06-27T21:42:43.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-06-27T21:42:44.083 INFO:tasks.workunit.client.0.smithi042.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-06-27T21:42:44.101 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-06-27T21:42:44.840 INFO:tasks.workunit.client.0.smithi042.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-06-27T21:42:44.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-06-27T21:42:45.416 INFO:tasks.workunit.client.0.smithi042.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-06-27T21:42:45.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-06-27T21:42:45.997 INFO:tasks.workunit.client.0.smithi042.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-06-27T21:42:46.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-06-27T21:42:46.613 INFO:tasks.workunit.client.0.smithi042.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-06-27T21:42:46.629 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-06-27T21:42:47.174 INFO:tasks.workunit.client.0.smithi042.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-06-27T21:42:47.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-06-27T21:42:47.945 INFO:tasks.workunit.client.0.smithi042.stderr:rule myrule already exists 2024-06-27T21:42:47.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-06-27T21:42:47.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-06-27T21:42:48.538 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'foo' already exists 2024-06-27T21:42:48.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T21:42:49.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-06-27T21:42:49.755 INFO:tasks.workunit.client.0.smithi042.stdout:hints = 1 2024-06-27T21:42:49.755 INFO:tasks.workunit.client.0.smithi042.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-06-27T21:42:49.755 INFO:tasks.workunit.client.0.smithi042.stdout:Object prefix: benchmark_data_smithi042_46833 2024-06-27T21:42:49.755 INFO:tasks.workunit.client.0.smithi042.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:42:49.755 INFO:tasks.workunit.client.0.smithi042.stdout: 0 0 0 0 0 0 - 0 2024-06-27T21:42:50.755 INFO:tasks.workunit.client.0.smithi042.stdout: 1 16 38 22 0.0859274 0.0859375 0.30819 0.559704 2024-06-27T21:42:51.755 INFO:tasks.workunit.client.0.smithi042.stdout: 2 16 96 80 0.156228 0.226562 0.331718 0.385685 2024-06-27T21:42:52.755 INFO:tasks.workunit.client.0.smithi042.stdout: 3 16 150 134 0.174454 0.210938 0.2914 0.345604 2024-06-27T21:42:53.756 INFO:tasks.workunit.client.0.smithi042.stdout: 4 16 204 188 0.183569 0.210938 0.314984 0.323197 2024-06-27T21:42:54.756 INFO:tasks.workunit.client.0.smithi042.stdout: 5 16 265 249 0.194505 0.238281 0.302567 0.312133 2024-06-27T21:42:55.756 INFO:tasks.workunit.client.0.smithi042.stdout: 6 16 326 310 0.201796 0.238281 0.266263 0.306272 2024-06-27T21:42:56.756 INFO:tasks.workunit.client.0.smithi042.stdout: 7 16 374 358 0.19975 0.1875 0.377757 0.307718 2024-06-27T21:42:57.756 INFO:tasks.workunit.client.0.smithi042.stdout: 8 16 425 409 0.19968 0.199219 0.285276 0.304751 2024-06-27T21:42:58.756 INFO:tasks.workunit.client.0.smithi042.stdout: 9 16 486 470 0.203964 0.238281 0.284724 0.300785 2024-06-27T21:42:59.756 INFO:tasks.workunit.client.0.smithi042.stdout: 10 16 553 537 0.209736 0.261719 0.250131 0.294067 2024-06-27T21:43:00.757 INFO:tasks.workunit.client.0.smithi042.stdout:Total time run: 10.1681 2024-06-27T21:43:00.757 INFO:tasks.workunit.client.0.smithi042.stdout:Total writes made: 553 2024-06-27T21:43:00.757 INFO:tasks.workunit.client.0.smithi042.stdout:Write size: 4096 2024-06-27T21:43:00.757 INFO:tasks.workunit.client.0.smithi042.stdout:Object size: 4096 2024-06-27T21:43:00.757 INFO:tasks.workunit.client.0.smithi042.stdout:Bandwidth (MB/sec): 0.212444 2024-06-27T21:43:00.757 INFO:tasks.workunit.client.0.smithi042.stdout:Stddev Bandwidth: 0.0487212 2024-06-27T21:43:00.757 INFO:tasks.workunit.client.0.smithi042.stdout:Max bandwidth (MB/sec): 0.261719 2024-06-27T21:43:00.758 INFO:tasks.workunit.client.0.smithi042.stdout:Min bandwidth (MB/sec): 0.0859375 2024-06-27T21:43:00.758 INFO:tasks.workunit.client.0.smithi042.stdout:Average IOPS: 54 2024-06-27T21:43:00.758 INFO:tasks.workunit.client.0.smithi042.stdout:Stddev IOPS: 12.4726 2024-06-27T21:43:00.758 INFO:tasks.workunit.client.0.smithi042.stdout:Max IOPS: 67 2024-06-27T21:43:00.758 INFO:tasks.workunit.client.0.smithi042.stdout:Min IOPS: 22 2024-06-27T21:43:00.758 INFO:tasks.workunit.client.0.smithi042.stdout:Average Latency(s): 0.29201 2024-06-27T21:43:00.758 INFO:tasks.workunit.client.0.smithi042.stdout:Stddev Latency(s): 0.0752216 2024-06-27T21:43:00.758 INFO:tasks.workunit.client.0.smithi042.stdout:Max latency(s): 0.739723 2024-06-27T21:43:00.758 INFO:tasks.workunit.client.0.smithi042.stdout:Min latency(s): 0.180307 2024-06-27T21:43:00.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-06-27T21:43:10.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-06-27T21:43:10.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:43:10.849 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:43:11.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T21:43:11.339 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T21:43:11.340 INFO:tasks.workunit.client.0.smithi042.stderr:2' 2024-06-27T21:43:11.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:43:11.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:43:11.340 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:43:11.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836510 2024-06-27T21:43:11.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836510 2024-06-27T21:43:11.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510' 2024-06-27T21:43:11.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:43:11.518 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T21:43:11.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672988 2024-06-27T21:43:11.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672988 2024-06-27T21:43:11.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672988' 2024-06-27T21:43:11.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:43:11.671 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T21:43:11.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509452 2024-06-27T21:43:11.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509452 2024-06-27T21:43:11.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672988 2-64424509452' 2024-06-27T21:43:11.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:43:11.825 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836510 2024-06-27T21:43:11.825 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:43:11.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:43:11.827 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836510 2024-06-27T21:43:11.827 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:43:11.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836510 2024-06-27T21:43:11.829 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836510 2024-06-27T21:43:11.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-06-27T21:43:11.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:43:12.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836510 2024-06-27T21:43:12.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T21:43:13.319 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T21:43:13.319 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:43:13.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836510 2024-06-27T21:43:13.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:43:13.760 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672988 2024-06-27T21:43:13.761 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:43:13.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T21:43:13.763 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672988 2024-06-27T21:43:13.763 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:43:13.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672988 2024-06-27T21:43:13.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-06-27T21:43:13.765 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672988 2024-06-27T21:43:13.765 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T21:43:14.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672988 -lt 42949672988 2024-06-27T21:43:14.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:43:14.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509452 2024-06-27T21:43:14.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:43:14.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T21:43:14.206 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509452 2024-06-27T21:43:14.206 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:43:14.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509452 2024-06-27T21:43:14.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-06-27T21:43:14.208 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509452 2024-06-27T21:43:14.208 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T21:43:14.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509453 -lt 64424509452 2024-06-27T21:43:14.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-06-27T21:43:14.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-06-27T21:43:15.123 INFO:tasks.workunit.client.0.smithi042.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.595ms); Slow OSD heartbeats on front (longest 0.601ms); 1 pool(s) do not have an application enabled 2024-06-27T21:43:15.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-06-27T21:43:15.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-06-27T21:43:15.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-06-27T21:43:15.713 INFO:tasks.workunit.client.0.smithi042.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.595ms); Slow OSD heartbeats on front (longest 0.601ms); 1 pool(s) do not have an application enabled 2024-06-27T21:43:15.713 INFO:tasks.workunit.client.0.smithi042.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.595ms) 2024-06-27T21:43:15.713 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.595 msec 2024-06-27T21:43:15.713 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.569 msec possibly improving 2024-06-27T21:43:15.714 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.565 msec 2024-06-27T21:43:15.714 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.543 msec 2024-06-27T21:43:15.714 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.508 msec 2024-06-27T21:43:15.714 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.502 msec 2024-06-27T21:43:15.714 INFO:tasks.workunit.client.0.smithi042.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.601ms) 2024-06-27T21:43:15.714 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.601 msec 2024-06-27T21:43:15.714 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.593 msec 2024-06-27T21:43:15.714 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.580 msec 2024-06-27T21:43:15.718 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.573 msec 2024-06-27T21:43:15.718 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.544 msec 2024-06-27T21:43:15.719 INFO:tasks.workunit.client.0.smithi042.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.544 msec 2024-06-27T21:43:15.719 INFO:tasks.workunit.client.0.smithi042.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-06-27T21:43:15.719 INFO:tasks.workunit.client.0.smithi042.stdout: application not enabled on pool 'foo' 2024-06-27T21:43:15.719 INFO:tasks.workunit.client.0.smithi042.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-06-27T21:43:15.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-06-27T21:43:15.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-06-27T21:43:15.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-06-27T21:43:15.733 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-06-27T21:43:15.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-06-27T21:43:15.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-06-27T21:43:15.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:43:15.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-06-27T21:43:15.735 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:43:15.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:43:15.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:43:15.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:43:15.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:43:15.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:43:15.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:43:15.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:43:15.846 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:43:15.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:43:15.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:43:15.847 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:43:15.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:43:15.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:43:15.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:43:15.848 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:43:15.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:43:15.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:43:15.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-06-27T21:43:15.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:43:15.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:43:15.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:43:15.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34572 2024-06-27T21:43:15.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:43:15.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:43:15.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-06-27T21:43:15.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/network-ping 0 2024-06-27T21:43:15.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-06-27T21:43:15.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-27T21:43:15.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-06-27T21:43:15.871 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:43:15.871 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:43:15.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:43:15.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:43:15.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:43:15.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:43:15.872 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:43:15.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:43:15.872 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:43:15.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:43:15.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:43:15.873 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:43:15.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:43:15.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:43:15.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:43:15.875 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:43:15.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:43:15.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-27T21:43:15.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-06-27T21:43:15.877 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:43:15.877 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:43:15.877 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.34572 2024-06-27T21:43:15.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.34572 2024-06-27T21:43:15.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:43:15.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:43:15.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-06-27T21:43:15.880 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-27T21:43:15.880 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-27T21:43:15.891 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-06-27T21:43:15.891 DEBUG:teuthology.orchestra.run.smithi042:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-06-27T21:43:15.943 INFO:tasks.workunit.client.0.smithi042.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-27T21:43:15.948 INFO:tasks.workunit.client.0.smithi042.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-27T21:43:15.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T21:43:15.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T21:43:15.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/ok-to-stop 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-06-27T21:43:15.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-06-27T21:43:15.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-06-27T21:43:15.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-06-27T21:43:15.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-06-27T21:43:15.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-06-27T21:43:15.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-06-27T21:43:15.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-06-27T21:43:15.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-06-27T21:43:15.950 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-06-27T21:43:15.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none ' 2024-06-27T21:43:15.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none ' 2024-06-27T21:43:15.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none ' 2024-06-27T21:43:15.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-06-27T21:43:15.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-27T21:43:15.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_0_osd 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_1_mon_checks 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_2_mons_checks 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_3_mons_checks 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_4_mons_checks 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_5_mons_checks' 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:43:15.953 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:43:15.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:43:15.954 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:43:15.954 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:43:15.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:43:15.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:43:15.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:43:15.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:43:15.955 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:43:15.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:43:15.955 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:43:15.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:43:15.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:43:15.956 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:43:15.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:43:15.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:43:15.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:43:15.957 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:43:15.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:43:15.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:43:15.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:43:15.958 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:43:15.959 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:43:15.959 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:43:15.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:43:15.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:43:15.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:43:15.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-27T21:43:15.960 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:43:15.960 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:43:15.960 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:43:15.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47167 2024-06-27T21:43:15.962 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:43:15.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-27T21:43:15.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-27T21:43:15.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:43:15.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-27T21:43:15.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-06-27T21:43:15.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-06-27T21:43:15.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-27T21:43:15.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-27T21:43:15.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:43:15.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:43:15.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:43:15.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:43:15.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-27T21:43:15.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-27T21:43:16.507 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:43:16.507 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:43:16.507 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:43:16.508 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:43:16.508 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:43:16.508 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:43:16.508 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:43:16.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-27T21:43:16.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:43:16.726 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:43:16.726 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:43:16.727 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:43:16.727 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:43:16.727 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:43:16.728 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:43:16.728 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:43:16.728 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:43:16.728 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:43:16.728 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:43:16.728 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:43:16.728 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:43:16.728 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:43:16.728 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get fsid 2024-06-27T21:43:16.846 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:43:16.846 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:43:16.846 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:43:16.846 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:43:16.847 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:43:16.847 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:43:16.847 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:43:16.847 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:43:16.847 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:43:16.848 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:43:16.848 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:43:16.848 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:43:16.848 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:43:16.848 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get mon_host 2024-06-27T21:43:16.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-06-27T21:43:16.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-06-27T21:43:16.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:43:16.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:43:16.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:43:16.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-06-27T21:43:16.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:43:17.875 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:43:17.876 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:43:17.876 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:43:17.876 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:43:17.876 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:43:17.876 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:43:17.876 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:43:17.877 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:43:17.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:43:17.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:43:17.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-06-27T21:43:17.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:43:17.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:43:17.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:43:17.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:43:17.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:43:17.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:43:17.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-06-27T21:43:17.918 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:43:17.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=86c1e8fc-e9ff-4f2b-b10f-b477fcc2dcf2 2024-06-27T21:43:17.919 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 86c1e8fc-e9ff-4f2b-b10f-b477fcc2dcf2 2024-06-27T21:43:17.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 86c1e8fc-e9ff-4f2b-b10f-b477fcc2dcf2' 2024-06-27T21:43:17.919 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:43:17.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD13H1m3YCFNxAAUtVfOtbjG8rpuxaLETMEsw== 2024-06-27T21:43:17.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD13H1m3YCFNxAAUtVfOtbjG8rpuxaLETMEsw=="}' 2024-06-27T21:43:17.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 86c1e8fc-e9ff-4f2b-b10f-b477fcc2dcf2 -i td/ok-to-stop/0/new.json 2024-06-27T21:43:18.264 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:43:18.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-06-27T21:43:18.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD13H1m3YCFNxAAUtVfOtbjG8rpuxaLETMEsw== --osd-uuid 86c1e8fc-e9ff-4f2b-b10f-b477fcc2dcf2 2024-06-27T21:43:18.313 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:18.307+0000 7fc356c077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:43:18.341 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:18.335+0000 7fc356c077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:43:18.342 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:18.339+0000 7fc356c077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:43:18.343 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:18.339+0000 7fc356c077c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-27T21:43:18.343 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:18.339+0000 7fc356c077c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-27T21:43:18.343 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:18.339+0000 7fc356c077c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-27T21:43:18.382 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:18.375+0000 7fc356c077c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-06-27T21:43:24.993 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-06-27T21:43:24.993 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:43:24.994 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:43:24.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:43:24.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:43:25.595 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:43:25.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:43:25.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:43:25.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:43:25.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:43:25.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:43:25.625 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:25.619+0000 7f8b3402a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:43:25.630 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:25.627+0000 7f8b3402a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:43:25.631 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:25.627+0000 7f8b3402a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:43:26.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:43:26.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:43:26.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:43:26.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:43:26.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:43:26.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:26.014 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:43:26.014 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:43:26.014 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:26.014 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:26.216 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:26.211+0000 7f8b3402a7c0 -1 Falling back to public interface 2024-06-27T21:43:26.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:27.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:27.451 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:43:27.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:27.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:43:27.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:27.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:27.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:28.788 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:43:28.783+0000 7f8b3402a7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:43:28.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:28.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:28.899 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:43:28.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:43:28.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:28.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:29.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:30.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:30.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:30.370 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:43:30.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:43:30.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:30.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:30.904 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:31.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:31.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:31.906 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:43:31.907 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:43:31.907 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:31.907 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:32.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:33.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:33.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:33.338 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:43:33.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:43:33.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:33.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:33.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:34.839 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:34.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:34.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:43:34.840 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:43:34.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:34.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:35.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:36.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:36.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:36.304 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:43:36.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:43:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:36.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:37.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:37.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:37.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:43:37.871 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:43:37.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:37.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:38.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:39.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:39.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:39.336 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:43:39.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:43:39.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:39.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:39.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:40.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:40.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:40.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:43:40.843 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:43:40.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:40.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:41.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:42.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:42.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:42.280 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:43:42.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:43:42.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:42.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:43.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:44.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:44.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:44.113 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:43:44.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:43:44.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:44.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:45.258 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:46.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:46.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:46.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:43:46.261 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:43:46.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:46.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:47.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:48.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:48.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:48.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:43:48.128 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:43:48.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:48.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:49.106 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:50.106 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:50.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:50.107 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:43:50.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:43:50.108 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:50.108 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:51.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:52.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:52.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:52.144 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:43:52.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:43:52.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:52.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:52.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:43:53.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:43:53.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:43:53.812 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:43:53.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:43:53.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:43:53.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:43:54.254 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4197604942,v1:127.0.0.1:6803/4197604942] [v2:127.0.0.1:6804/4197604942,v1:127.0.0.1:6805/4197604942] exists,up 86c1e8fc-e9ff-4f2b-b10f-b477fcc2dcf2 2024-06-27T21:43:54.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:43:54.256 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:43:54.257 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:43:54.257 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:43:54.257 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:43:54.257 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-06-27T21:43:54.323 INFO:tasks.workunit.client.0.smithi042.stdout:starting mds.a at 2024-06-27T21:43:54.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-06-27T21:43:55.058 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'meta' already exists 2024-06-27T21:43:55.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-06-27T21:43:55.726 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'data' already exists 2024-06-27T21:43:55.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-06-27T21:43:56.909 INFO:tasks.workunit.client.0.smithi042.stderr:filesystem 'myfs' already exists 2024-06-27T21:43:56.929 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-06-27T21:44:01.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-06-27T21:44:02.314 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-06-27T21:44:02.318 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-06-27T21:44:02.732 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-06-27T21:44:02.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-06-27T21:44:03.327 INFO:tasks.workunit.client.0.smithi042.stderr:should be safe to stop [dne] 2024-06-27T21:44:03.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-06-27T21:44:03.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-06-27T21:44:03.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-06-27T21:44:03.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2024-06-27T21:44:03.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-06-27T21:44:03.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2024-06-27T21:44:03.344 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-06-27T21:44:03.344 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:44:03.344 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:44:03.345 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:44:03.345 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:44:03.345 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:44:03.345 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:44:03.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-06-27T21:44:03.377 INFO:tasks.workunit.client.0.smithi042.stdout:starting mds.b at 2024-06-27T21:44:03.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-06-27T21:44:08.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-06-27T21:44:08.939 INFO:tasks.workunit.client.0.smithi042.stderr:should be safe to stop [a] 2024-06-27T21:44:08.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-06-27T21:44:09.543 INFO:tasks.workunit.client.0.smithi042.stderr:should be safe to stop [b] 2024-06-27T21:44:09.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-06-27T21:44:09.933 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-06-27T21:44:09.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-06-27T21:44:10.548 INFO:tasks.workunit.client.0.smithi042.stderr:should be safe to stop [a,dne1,dne2] 2024-06-27T21:44:10.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-06-27T21:44:11.122 INFO:tasks.workunit.client.0.smithi042.stderr:should be safe to stop [b,dne] 2024-06-27T21:44:11.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-06-27T21:44:11.529 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-06-27T21:44:11.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-06-27T21:44:12.122 INFO:tasks.workunit.client.0.smithi042.stderr:should be safe to stop [dne1,dne2] 2024-06-27T21:44:12.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-06-27T21:44:12.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:44:12.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:44:12.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:44:12.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:44:12.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:44:12.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:44:12.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-27T21:44:12.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:44:12.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:44:12.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:44:12.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:44:12.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:44:12.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:44:12.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-27T21:44:12.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:44:12.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:44:12.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:44:12.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:44:12.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:44:12.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:44:12.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:44:12.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:44:12.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:44:12.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:44:12.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:44:12.364 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:44:12.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:44:12.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:44:12.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:44:12.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:44:12.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:44:12.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:44:12.366 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:44:12.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:44:12.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:44:12.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:44:12.375 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:44:12.375 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:44:12.375 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:44:12.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:44:12.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:44:12.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:44:12.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-27T21:44:12.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-27T21:44:12.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-27T21:44:12.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-27T21:44:12.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:44:12.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:44:12.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:44:12.378 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:44:12.378 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:44:12.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:44:12.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:44:12.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:44:12.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:44:12.379 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:44:12.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:44:12.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:44:12.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:44:12.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:44:12.381 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:44:12.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:44:12.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:44:12.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:44:12.382 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:44:12.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:44:12.384 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:44:12.384 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:44:12.385 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:44:12.385 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:44:12.386 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:44:12.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:44:12.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:44:12.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:44:12.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-27T21:44:12.388 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:44:12.388 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:44:12.388 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:44:12.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47167 2024-06-27T21:44:12.390 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:44:12.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:44:12.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:44:12.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-27T21:44:12.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-06-27T21:44:12.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-06-27T21:44:12.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-27T21:44:12.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-27T21:44:12.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:44:12.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:44:12.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:44:12.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:44:12.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-27T21:44:12.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-27T21:44:12.867 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:44:12.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:44:12.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:44:12.868 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:44:12.868 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:44:12.868 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:44:12.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:44:12.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-27T21:44:13.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:44:13.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:44:13.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:44:13.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:44:13.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:44:13.080 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:44:13.080 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:44:13.080 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:44:13.080 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:44:13.080 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:44:13.080 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:44:13.080 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:44:13.080 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:44:13.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:44:13.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get fsid 2024-06-27T21:44:13.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:44:13.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:44:13.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:44:13.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:44:13.202 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:44:13.202 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:44:13.202 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:44:13.202 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:44:13.203 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:44:13.203 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:44:13.203 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:44:13.203 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:44:13.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:44:13.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get mon_host 2024-06-27T21:44:13.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-06-27T21:44:13.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-06-27T21:44:13.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:44:13.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:44:13.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:44:13.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-06-27T21:44:13.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:44:14.258 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:44:14.258 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:44:14.258 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:44:14.258 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:44:14.258 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:44:14.258 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:44:14.258 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:44:14.259 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:44:14.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:44:14.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-06-27T21:44:14.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-06-27T21:44:14.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:44:14.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:44:14.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:44:14.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-06-27T21:44:14.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-27T21:44:14.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:44:14.294 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:44:14.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:44:14.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-06-27T21:44:14.296 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:44:14.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ec43247d-de70-46e3-a030-1afc2de6a4ff 2024-06-27T21:44:14.297 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 ec43247d-de70-46e3-a030-1afc2de6a4ff 2024-06-27T21:44:14.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ec43247d-de70-46e3-a030-1afc2de6a4ff' 2024-06-27T21:44:14.297 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:44:14.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAu3X1mP4FdEhAAQ4T7H17DkFYm1sGfR3MsHA== 2024-06-27T21:44:14.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAu3X1mP4FdEhAAQ4T7H17DkFYm1sGfR3MsHA=="}' 2024-06-27T21:44:14.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ec43247d-de70-46e3-a030-1afc2de6a4ff -i td/ok-to-stop/0/new.json 2024-06-27T21:44:14.632 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:44:14.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-06-27T21:44:14.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAu3X1mP4FdEhAAQ4T7H17DkFYm1sGfR3MsHA== --osd-uuid ec43247d-de70-46e3-a030-1afc2de6a4ff 2024-06-27T21:44:14.683 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:14.679+0000 7fe74309d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:14.718 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:14.711+0000 7fe74309d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:14.719 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:14.715+0000 7fe74309d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:14.719 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:14.715+0000 7fe74309d7c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-27T21:44:14.719 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:14.715+0000 7fe74309d7c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-27T21:44:14.719 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:14.715+0000 7fe74309d7c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-06-27T21:44:14.757 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:14.751+0000 7fe74309d7c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-06-27T21:44:21.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-06-27T21:44:21.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:44:21.305 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:44:21.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:44:21.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:44:21.893 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:44:21.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:44:21.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:44:21.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:44:21.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:44:21.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:44:21.924 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:21.919+0000 7f127f7b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:21.932 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:21.927+0000 7f127f7b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:21.934 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:21.927+0000 7f127f7b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:22.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:44:22.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:44:22.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:44:22.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:44:22.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:44:22.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:22.327 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:44:22.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:44:22.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:22.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:22.501 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:22.495+0000 7f127f7b07c0 -1 Falling back to public interface 2024-06-27T21:44:22.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:23.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:23.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:23.760 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:44:23.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:44:23.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:23.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:24.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:25.095 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:25.091+0000 7f127f7b07c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:44:25.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:25.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:25.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:44:25.190 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:44:25.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:25.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:25.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:26.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:26.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:26.732 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:44:26.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:44:26.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:26.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:27.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:28.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:28.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:28.178 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:44:28.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:44:28.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:28.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:28.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:29.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:29.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:29.631 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:44:29.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:44:29.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:29.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:30.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:31.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:31.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:31.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:44:31.150 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:44:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:31.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:32.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:32.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:32.591 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:44:32.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:44:32.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:32.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:33.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:34.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:34.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:34.080 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:44:34.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:44:34.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:34.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:34.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:35.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:35.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:35.527 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:44:35.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:44:35.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:35.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:36.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:37.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:37.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:37.024 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:44:37.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:44:37.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:37.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:37.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:38.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:38.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:38.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:44:38.510 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:44:38.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:38.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:39.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:40.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:40.489 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:40.489 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:44:40.489 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:44:40.490 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:40.490 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:41.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:42.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:42.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:42.417 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:44:42.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:44:42.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:42.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:43.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:44.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:44.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:44.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:44:44.252 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:44:44.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:44.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:45.101 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:46.102 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:46.103 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:46.103 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:44:46.103 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:44:46.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:46.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:47.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:48.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:48.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:48.193 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:44:48.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:44:48.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:48.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:48.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:44:49.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:44:49.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:49.998 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:44:49.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:44:49.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:49.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:44:50.445 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3090419904,v1:127.0.0.1:6803/3090419904] [v2:127.0.0.1:6804/3090419904,v1:127.0.0.1:6805/3090419904] exists,up ec43247d-de70-46e3-a030-1afc2de6a4ff 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:44:50.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:44:50.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:44:50.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-06-27T21:44:50.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-06-27T21:44:50.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:44:50.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:44:50.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-06-27T21:44:50.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:44:50.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:44:50.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:44:50.449 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:44:50.449 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:44:50.450 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:44:50.450 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:44:50.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:44:50.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:44:50.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:44:50.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:44:50.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-06-27T21:44:50.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-06-27T21:44:50.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:44:50.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:44:50.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:44:50.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:44:50.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:44:50.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:44:50.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2024-06-27T21:44:50.451 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:44:50.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=17897c1b-dd04-4c57-b727-63e7fe96bf1c 2024-06-27T21:44:50.452 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 17897c1b-dd04-4c57-b727-63e7fe96bf1c 2024-06-27T21:44:50.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 17897c1b-dd04-4c57-b727-63e7fe96bf1c' 2024-06-27T21:44:50.452 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:44:50.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBS3X1m9zHcGxAAwOKx1qCkn3mIHgsAGNiDpg== 2024-06-27T21:44:50.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBS3X1m9zHcGxAAwOKx1qCkn3mIHgsAGNiDpg=="}' 2024-06-27T21:44:50.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 17897c1b-dd04-4c57-b727-63e7fe96bf1c -i td/ok-to-stop/1/new.json 2024-06-27T21:44:50.947 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:44:50.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2024-06-27T21:44:50.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBS3X1m9zHcGxAAwOKx1qCkn3mIHgsAGNiDpg== --osd-uuid 17897c1b-dd04-4c57-b727-63e7fe96bf1c 2024-06-27T21:44:50.997 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:50.990+0000 7f635a4827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:50.999 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:50.994+0000 7f635a4827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:51.000 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:50.994+0000 7f635a4827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:51.000 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:50.994+0000 7f635a4827c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-06-27T21:44:51.001 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:50.994+0000 7f635a4827c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-06-27T21:44:51.001 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:50.994+0000 7f635a4827c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-06-27T21:44:51.038 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:51.030+0000 7f635a4827c0 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-06-27T21:44:57.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-06-27T21:44:57.689 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:44:57.690 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T21:44:57.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T21:44:57.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:44:58.265 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T21:44:58.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T21:44:58.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:44:58.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:44:58.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:44:58.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:44:58.296 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:58.290+0000 7f93a1dfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:58.310 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:58.306+0000 7f93a1dfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:58.312 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:58.306+0000 7f93a1dfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:44:58.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T21:44:58.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:44:58.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T21:44:58.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:44:58.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:44:58.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:44:58.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:44:58.735 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:44:58.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:44:58.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:44:58.900 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:44:58.894+0000 7f93a1dfa7c0 -1 Falling back to public interface 2024-06-27T21:44:59.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:00.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:00.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:00.228 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:45:00.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:45:00.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:00.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:00.719 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:01.620 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:01.614+0000 7f93a1dfa7c0 -1 osd.1 0 log_to_monitors true 2024-06-27T21:45:01.720 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:01.720 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:01.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:45:01.721 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:45:01.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:01.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:02.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:03.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:03.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:03.345 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:45:03.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:45:03.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:03.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:04.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:05.005 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:05.005 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:05.005 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:45:05.006 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:45:05.006 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:05.006 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:05.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:06.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:06.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:06.677 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:45:06.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:45:06.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:06.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:07.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:08.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:08.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:08.125 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:45:08.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:45:08.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:08.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:08.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:09.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:09.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:09.603 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:45:09.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:45:09.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:09.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:10.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:11.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:11.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:11.086 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:45:11.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:45:11.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:11.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:11.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:12.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:12.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:12.668 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:45:12.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:45:12.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:12.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:13.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:14.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:14.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:14.130 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:45:14.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:45:14.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:14.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:14.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:15.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:15.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:15.611 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:45:15.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:45:15.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:15.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:16.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:17.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:17.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:17.709 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:45:17.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:45:17.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:17.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:18.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:19.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:19.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:19.869 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:45:19.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:45:19.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:19.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:20.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:21.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:21.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:21.675 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:45:21.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:45:21.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:21.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:22.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:23.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:23.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:23.671 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:45:23.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:45:23.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:23.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:24.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:25.518 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:45:25.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:45:25.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:25.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:26.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:27.474 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:27.474 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:27.474 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:45:27.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:45:27.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:27.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:45:27.975 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1765046652,v1:127.0.0.1:6811/1765046652] [v2:127.0.0.1:6812/1765046652,v1:127.0.0.1:6813/1765046652] exists,up 17897c1b-dd04-4c57-b727-63e7fe96bf1c 2024-06-27T21:45:27.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:45:27.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:45:27.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:45:27.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-06-27T21:45:27.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-06-27T21:45:27.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:45:27.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-27T21:45:27.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:45:27.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2024-06-27T21:45:27.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-27T21:45:27.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:45:27.980 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:45:27.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:45:27.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2024-06-27T21:45:27.982 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:45:27.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bba0925c-27eb-44d1-81c3-e276c4dc75fa 2024-06-27T21:45:27.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bba0925c-27eb-44d1-81c3-e276c4dc75fa' 2024-06-27T21:45:27.983 INFO:tasks.workunit.client.0.smithi042.stdout:add osd2 bba0925c-27eb-44d1-81c3-e276c4dc75fa 2024-06-27T21:45:27.983 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:45:27.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB33X1mCEhSOxAA72YCU79HWkh5hOc1y0WiAQ== 2024-06-27T21:45:27.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB33X1mCEhSOxAA72YCU79HWkh5hOc1y0WiAQ=="}' 2024-06-27T21:45:27.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bba0925c-27eb-44d1-81c3-e276c4dc75fa -i td/ok-to-stop/2/new.json 2024-06-27T21:45:28.513 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:45:28.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2024-06-27T21:45:28.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB33X1mCEhSOxAA72YCU79HWkh5hOc1y0WiAQ== --osd-uuid bba0925c-27eb-44d1-81c3-e276c4dc75fa 2024-06-27T21:45:28.558 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:28.550+0000 7f132f46b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:45:28.560 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:28.554+0000 7f132f46b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:45:28.561 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:28.554+0000 7f132f46b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:45:28.562 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:28.554+0000 7f132f46b7c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-06-27T21:45:28.562 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:28.554+0000 7f132f46b7c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-06-27T21:45:28.562 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:28.554+0000 7f132f46b7c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-06-27T21:45:28.621 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:28.614+0000 7f132f46b7c0 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-06-27T21:45:35.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-06-27T21:45:35.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:45:35.284 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd2 key to auth repository 2024-06-27T21:45:35.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-27T21:45:35.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:45:35.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-27T21:45:35.866 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T21:45:35.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:45:35.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:45:35.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:45:35.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:45:35.896 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:35.890+0000 7fcdb8d667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:45:35.939 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:35.934+0000 7fcdb8d667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:45:35.940 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:35.934+0000 7fcdb8d667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:45:36.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-27T21:45:36.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:45:36.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T21:45:36.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:45:36.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:45:36.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:36.380 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:45:36.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:45:36.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:36.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:36.517 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:36.510+0000 7fcdb8d667c0 -1 Falling back to public interface 2024-06-27T21:45:36.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:37.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:37.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:37.830 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:45:37.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:45:37.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:37.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:38.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:39.158 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:45:39.150+0000 7fcdb8d667c0 -1 osd.2 0 log_to_monitors true 2024-06-27T21:45:39.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:39.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:39.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:45:39.273 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:45:39.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:39.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:39.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:40.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:40.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:40.898 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:45:40.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:45:40.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:40.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:41.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:42.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:42.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:42.379 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:45:42.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:45:42.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:42.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:42.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:43.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:43.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:43.819 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:45:43.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:45:43.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:43.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:44.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:45.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:45.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:45.433 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:45:45.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:45:45.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:45.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:45.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:46.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:46.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:46.883 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:45:46.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:45:46.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:46.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:47.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:48.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:48.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:48.344 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:45:48.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:45:48.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:48.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:48.785 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:49.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:49.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:49.787 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:45:49.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:45:49.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:49.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:50.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:51.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:51.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:51.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:45:51.389 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:45:51.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:51.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:51.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:52.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:52.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:52.875 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:45:52.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:45:52.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:52.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:53.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:54.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:54.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:54.325 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:45:54.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:45:54.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:54.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:54.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:55.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:55.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:55.911 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:45:55.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:45:55.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:55.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:56.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:57.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:57.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:57.990 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:45:57.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:45:57.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:57.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:45:58.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:45:59.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:45:59.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:45:59.896 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:45:59.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:45:59.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:45:59.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:46:00.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:01.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:01.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:01.485 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:46:01.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:46:01.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:01.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:46:01.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:02.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:02.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:02.990 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:46:02.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:46:02.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:02.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:46:03.746 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:04.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:04.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:04.749 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T21:46:04.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T21:46:04.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:04.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:46:05.195 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4001950973,v1:127.0.0.1:6819/4001950973] [v2:127.0.0.1:6820/4001950973,v1:127.0.0.1:6821/4001950973] exists,up bba0925c-27eb-44d1-81c3-e276c4dc75fa 2024-06-27T21:46:05.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:46:05.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:46:05.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:46:05.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-06-27T21:46:05.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-06-27T21:46:05.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:46:05.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-06-27T21:46:05.199 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-06-27T21:46:05.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:46:05.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:46:05.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:46:05.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:46:05.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:46:05.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:46:05.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2024-06-27T21:46:05.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:46:05.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9eb6e07f-6da2-47d6-9dad-3ed73c9ff465 2024-06-27T21:46:05.202 INFO:tasks.workunit.client.0.smithi042.stdout:add osd3 9eb6e07f-6da2-47d6-9dad-3ed73c9ff465 2024-06-27T21:46:05.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9eb6e07f-6da2-47d6-9dad-3ed73c9ff465' 2024-06-27T21:46:05.202 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:46:05.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCd3X1mI1rsDBAAE7LVvpLmPbTvjb1YkOm3lQ== 2024-06-27T21:46:05.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCd3X1mI1rsDBAAE7LVvpLmPbTvjb1YkOm3lQ=="}' 2024-06-27T21:46:05.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9eb6e07f-6da2-47d6-9dad-3ed73c9ff465 -i td/ok-to-stop/3/new.json 2024-06-27T21:46:05.751 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:46:05.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2024-06-27T21:46:05.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCd3X1mI1rsDBAAE7LVvpLmPbTvjb1YkOm3lQ== --osd-uuid 9eb6e07f-6da2-47d6-9dad-3ed73c9ff465 2024-06-27T21:46:05.805 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:05.798+0000 7fdedc3d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:46:05.807 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:05.802+0000 7fdedc3d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:46:05.809 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:05.802+0000 7fdedc3d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:46:05.809 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:05.802+0000 7fdedc3d97c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-06-27T21:46:05.809 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:05.802+0000 7fdedc3d97c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-06-27T21:46:05.809 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:05.802+0000 7fdedc3d97c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-06-27T21:46:05.893 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:05.886+0000 7fdedc3d97c0 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-06-27T21:46:12.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-06-27T21:46:12.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:46:12.656 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd3 key to auth repository 2024-06-27T21:46:12.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-27T21:46:12.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:46:13.250 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.3 2024-06-27T21:46:13.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-27T21:46:13.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:46:13.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:46:13.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:46:13.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:46:13.274 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:13.270+0000 7f6018d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:46:13.310 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:13.302+0000 7f6018d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:46:13.311 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:13.306+0000 7f6018d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:46:13.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-27T21:46:13.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:46:13.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-27T21:46:13.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:46:13.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:46:13.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:13.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:46:13.741 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:46:13.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:13.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:13.901 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:13.894+0000 7f6018d0a7c0 -1 Falling back to public interface 2024-06-27T21:46:14.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:15.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:15.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:15.200 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:46:15.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:46:15.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:15.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:15.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:16.563 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:46:16.558+0000 7f6018d0a7c0 -1 osd.3 0 log_to_monitors true 2024-06-27T21:46:16.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:16.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:16.672 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:46:16.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:46:16.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:16.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:17.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:18.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:18.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:18.246 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:46:18.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:46:18.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:18.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:18.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:19.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:19.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:19.780 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:46:19.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:46:19.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:19.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:20.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:21.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:21.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:21.229 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:46:21.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:46:21.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:21.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:21.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:22.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:22.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:22.852 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:46:22.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:46:22.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:22.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:23.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:24.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:24.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:24.503 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:46:24.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:46:24.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:24.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:24.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:25.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:25.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:25.954 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:46:25.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:46:25.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:25.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:26.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:27.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:27.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:27.517 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:46:27.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:46:27.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:27.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:28.005 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:29.007 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:29.007 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:29.007 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:46:29.007 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:46:29.008 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:29.008 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:29.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:30.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:30.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:30.551 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:46:30.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:46:30.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:30.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:31.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:32.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:32.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:32.220 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:46:32.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:46:32.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:32.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:33.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:34.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:34.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:34.144 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:46:34.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:46:34.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:34.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:34.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:35.993 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:35.993 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:35.994 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:46:35.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:46:35.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:35.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:36.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:37.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:37.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:37.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:46:37.843 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:46:37.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:37.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:38.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:39.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:39.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:39.870 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:46:39.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:46:39.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:39.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:40.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:41.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:41.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:41.602 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:46:41.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:46:41.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:41.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:42.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:46:43.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:46:43.048 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:46:43.049 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T21:46:43.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T21:46:43.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:46:43.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-27T21:46:43.560 INFO:tasks.workunit.client.0.smithi042.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1711402949,v1:127.0.0.1:6827/1711402949] [v2:127.0.0.1:6828/1711402949,v1:127.0.0.1:6829/1711402949] exists,up 9eb6e07f-6da2-47d6-9dad-3ed73c9ff465 2024-06-27T21:46:43.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:46:43.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:46:43.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:46:43.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-06-27T21:46:44.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-06-27T21:46:45.157 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'ec' already exists 2024-06-27T21:46:45.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-06-27T21:46:45.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T21:46:45.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T21:46:45.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T21:46:45.221 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T21:46:45.221 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T21:46:45.221 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T21:46:45.221 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T21:46:45.221 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T21:46:45.221 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T21:46:45.275 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T21:46:45.275 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T21:46:45.276 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T21:46:45.276 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T21:46:45.276 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:46:45.276 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:46:46.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T21:46:46.035 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T21:46:46.035 INFO:tasks.workunit.client.0.smithi042.stderr:2 2024-06-27T21:46:46.035 INFO:tasks.workunit.client.0.smithi042.stderr:3' 2024-06-27T21:46:46.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:46:46.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:46:46.036 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:46:46.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836509 2024-06-27T21:46:46.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836509 2024-06-27T21:46:46.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509' 2024-06-27T21:46:46.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:46:46.292 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T21:46:46.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-06-27T21:46:46.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-06-27T21:46:46.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672985' 2024-06-27T21:46:46.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:46:46.644 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T21:46:46.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-06-27T21:46:46.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-06-27T21:46:46.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672985 2-64424509455' 2024-06-27T21:46:46.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:46:46.838 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-27T21:46:47.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345929 2024-06-27T21:46:47.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345929 2024-06-27T21:46:47.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672985 2-64424509455 3-85899345929' 2024-06-27T21:46:47.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:46:47.124 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836509 2024-06-27T21:46:47.124 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:46:47.124 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:46:47.125 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836509 2024-06-27T21:46:47.125 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:46:47.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836509 2024-06-27T21:46:47.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-06-27T21:46:47.127 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836509 2024-06-27T21:46:47.127 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:46:47.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836509 2024-06-27T21:46:47.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T21:46:48.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T21:46:48.727 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:46:49.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836509 2024-06-27T21:46:49.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:46:49.168 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-06-27T21:46:49.169 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:46:49.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T21:46:49.170 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-06-27T21:46:49.171 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:46:49.171 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672985 2024-06-27T21:46:49.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-06-27T21:46:49.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-06-27T21:46:49.172 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T21:46:49.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672986 -lt 42949672985 2024-06-27T21:46:49.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:46:49.636 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-06-27T21:46:49.636 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:46:49.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T21:46:49.639 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-06-27T21:46:49.639 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:46:49.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-06-27T21:46:49.641 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509455 2024-06-27T21:46:49.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-06-27T21:46:49.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T21:46:50.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509455 2024-06-27T21:46:50.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:46:50.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345929 2024-06-27T21:46:50.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:46:50.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-27T21:46:50.089 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345929 2024-06-27T21:46:50.089 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:46:50.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345929 2024-06-27T21:46:50.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-06-27T21:46:50.090 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.3 seq 85899345929 2024-06-27T21:46:50.090 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-27T21:46:50.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345930 -lt 85899345929 2024-06-27T21:46:50.516 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T21:46:50.516 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:46:50.516 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:46:51.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-06-27T21:46:51.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T21:46:51.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T21:46:51.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T21:46:51.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T21:46:51.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T21:46:51.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T21:46:51.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T21:46:51.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-06-27T21:46:51.547 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T21:46:51.547 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:46:51.547 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:46:52.102 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-06-27T21:46:52.102 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T21:46:52.102 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T21:46:52.103 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-06-27T21:46:53.109 INFO:tasks.workunit.client.0.smithi042.stderr:set pool 1 min_size to 3 2024-06-27T21:46:53.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-06-27T21:46:53.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T21:46:53.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T21:46:53.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T21:46:53.136 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T21:46:53.137 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T21:46:53.137 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T21:46:53.137 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T21:46:53.137 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T21:46:53.137 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T21:46:53.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T21:46:53.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T21:46:53.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T21:46:53.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T21:46:53.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:46:53.263 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:46:53.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T21:46:53.725 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T21:46:53.726 INFO:tasks.workunit.client.0.smithi042.stderr:2 2024-06-27T21:46:53.726 INFO:tasks.workunit.client.0.smithi042.stderr:3' 2024-06-27T21:46:53.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:46:53.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:46:53.726 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:46:53.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836512 2024-06-27T21:46:53.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836512 2024-06-27T21:46:53.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512' 2024-06-27T21:46:53.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:46:53.913 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T21:46:54.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672989 2024-06-27T21:46:54.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672989 2024-06-27T21:46:54.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672989' 2024-06-27T21:46:54.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:46:54.073 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T21:46:54.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509458 2024-06-27T21:46:54.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509458 2024-06-27T21:46:54.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672989 2-64424509458' 2024-06-27T21:46:54.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:46:54.223 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-27T21:46:54.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345933 2024-06-27T21:46:54.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345933 2024-06-27T21:46:54.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672989 2-64424509458 3-85899345933' 2024-06-27T21:46:54.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:46:54.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836512 2024-06-27T21:46:54.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:46:54.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:46:54.381 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:46:54.381 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836512 2024-06-27T21:46:54.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836512 2024-06-27T21:46:54.382 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836512 2024-06-27T21:46:54.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-06-27T21:46:54.383 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:46:54.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836512 2024-06-27T21:46:54.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T21:46:55.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T21:46:55.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:46:56.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836512 -lt 21474836512 2024-06-27T21:46:56.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:46:56.298 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672989 2024-06-27T21:46:56.298 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:46:56.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T21:46:56.300 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672989 2024-06-27T21:46:56.300 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:46:56.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672989 2024-06-27T21:46:56.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-06-27T21:46:56.303 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672989 2024-06-27T21:46:56.303 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T21:46:56.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672989 -lt 42949672989 2024-06-27T21:46:56.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:46:56.792 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509458 2024-06-27T21:46:56.792 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:46:56.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T21:46:56.795 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-06-27T21:46:56.795 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:46:56.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509458 2024-06-27T21:46:56.796 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509458 2024-06-27T21:46:56.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-06-27T21:46:56.797 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T21:46:57.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509458 2024-06-27T21:46:57.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:46:57.266 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345933 2024-06-27T21:46:57.266 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:46:57.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-27T21:46:57.268 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:46:57.269 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345933 2024-06-27T21:46:57.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345933 2024-06-27T21:46:57.270 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.3 seq 85899345933 2024-06-27T21:46:57.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-06-27T21:46:57.270 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-27T21:46:57.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345933 -lt 85899345933 2024-06-27T21:46:57.717 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T21:46:57.717 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:46:57.717 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:46:58.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-06-27T21:46:58.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T21:46:58.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T21:46:58.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T21:46:58.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T21:46:58.339 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T21:46:58.339 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T21:46:58.339 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T21:46:58.784 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-06-27T21:46:58.785 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T21:46:58.785 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:46:58.785 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:46:59.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-06-27T21:46:59.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T21:46:59.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T21:46:59.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-06-27T21:46:59.774 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:46:59.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-06-27T21:47:00.207 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:00.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-06-27T21:47:00.641 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:00.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-06-27T21:47:01.078 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:01.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-06-27T21:47:01.398 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:01.398 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-27T21:47:01.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-06-27T21:47:01.706 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:01.706 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-27T21:47:01.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-06-27T21:47:01.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-06-27T21:47:02.142 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:02.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-06-27T21:47:02.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-06-27T21:47:02.559 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:02.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-06-27T21:47:03.486 INFO:tasks.workunit.client.0.smithi042.stderr:set pool 1 min_size to 2 2024-06-27T21:47:03.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-06-27T21:47:03.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T21:47:03.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T21:47:03.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T21:47:03.511 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T21:47:03.512 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T21:47:03.512 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T21:47:03.512 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T21:47:03.512 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T21:47:03.512 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T21:47:03.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T21:47:03.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T21:47:03.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T21:47:03.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T21:47:03.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:47:03.636 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:47:04.438 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T21:47:04.438 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T21:47:04.439 INFO:tasks.workunit.client.0.smithi042.stderr:2 2024-06-27T21:47:04.439 INFO:tasks.workunit.client.0.smithi042.stderr:3' 2024-06-27T21:47:04.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:47:04.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:47:04.439 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:47:04.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836516 2024-06-27T21:47:04.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836516 2024-06-27T21:47:04.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516' 2024-06-27T21:47:04.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:47:04.602 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T21:47:04.765 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672993 2024-06-27T21:47:04.765 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672993 2024-06-27T21:47:04.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672993' 2024-06-27T21:47:04.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:47:04.766 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T21:47:04.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509462 2024-06-27T21:47:04.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509462 2024-06-27T21:47:04.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672993 2-64424509462' 2024-06-27T21:47:04.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:47:04.919 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-27T21:47:05.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345937 2024-06-27T21:47:05.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345937 2024-06-27T21:47:05.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672993 2-64424509462 3-85899345937' 2024-06-27T21:47:05.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:47:05.075 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836516 2024-06-27T21:47:05.075 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:47:05.075 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:47:05.076 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836516 2024-06-27T21:47:05.076 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:47:05.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836516 2024-06-27T21:47:05.077 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836516 2024-06-27T21:47:05.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-06-27T21:47:05.078 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:47:05.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836516 -lt 21474836516 2024-06-27T21:47:05.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:47:05.503 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:47:05.503 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672993 2024-06-27T21:47:05.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T21:47:05.505 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672993 2024-06-27T21:47:05.505 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:47:05.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672993 2024-06-27T21:47:05.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-06-27T21:47:05.507 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672993 2024-06-27T21:47:05.507 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T21:47:05.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672993 -lt 42949672993 2024-06-27T21:47:05.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:47:05.935 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509462 2024-06-27T21:47:05.935 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:47:05.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T21:47:05.937 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509462 2024-06-27T21:47:05.937 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:47:05.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509462 2024-06-27T21:47:05.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-06-27T21:47:05.939 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509462 2024-06-27T21:47:05.939 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T21:47:06.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509462 -lt 64424509462 2024-06-27T21:47:06.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:47:06.376 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345937 2024-06-27T21:47:06.376 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:47:06.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-27T21:47:06.379 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345937 2024-06-27T21:47:06.379 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:47:06.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345937 2024-06-27T21:47:06.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2024-06-27T21:47:06.380 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.3 seq 85899345937 2024-06-27T21:47:06.381 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-27T21:47:06.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345937 -lt 85899345937 2024-06-27T21:47:06.820 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T21:47:06.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:47:06.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:47:07.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-06-27T21:47:07.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T21:47:07.417 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T21:47:07.417 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T21:47:07.417 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T21:47:07.417 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T21:47:07.417 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T21:47:07.417 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T21:47:07.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-06-27T21:47:07.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T21:47:07.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:47:07.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:47:08.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-06-27T21:47:08.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T21:47:08.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T21:47:08.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-06-27T21:47:08.822 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:08.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-06-27T21:47:09.234 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:09.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-06-27T21:47:09.558 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:09.558 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-27T21:47:09.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-06-27T21:47:09.859 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:09.859 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-27T21:47:09.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-06-27T21:47:09.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-06-27T21:47:10.296 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:10.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-06-27T21:47:10.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-06-27T21:47:10.715 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:10.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-06-27T21:47:10.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-06-27T21:47:11.133 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:11.133 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-06-27T21:47:11.133 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-06-27T21:47:11.555 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:11.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-06-27T21:47:11.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:47:11.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:47:11.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:47:11.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:47:11.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:47:11.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:47:11.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-06-27T21:47:12.898 INFO:tasks.workunit.client.0.smithi042.stderr:osd.0 is already down. 2024-06-27T21:47:12.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-06-27T21:47:12.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-06-27T21:47:12.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-06-27T21:47:12.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-06-27T21:47:12.919 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-06-27T21:47:12.920 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T21:47:12.920 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T21:47:12.920 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T21:47:12.920 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T21:47:12.920 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T21:47:13.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T21:47:13.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-06-27T21:47:13.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-06-27T21:47:13.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-06-27T21:47:13.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:47:13.038 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:47:13.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T21:47:13.549 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T21:47:13.549 INFO:tasks.workunit.client.0.smithi042.stderr:2 2024-06-27T21:47:13.549 INFO:tasks.workunit.client.0.smithi042.stderr:3' 2024-06-27T21:47:13.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:47:13.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:47:13.549 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:47:13.664 INFO:tasks.workunit.client.0.smithi042.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-06-27T21:47:13.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-06-27T21:47:13.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-06-27T21:47:13.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-06-27T21:47:13.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:47:13.667 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T21:47:13.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672997 2024-06-27T21:47:13.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672997 2024-06-27T21:47:13.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672997' 2024-06-27T21:47:13.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:47:13.811 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T21:47:13.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509466 2024-06-27T21:47:13.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509466 2024-06-27T21:47:13.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672997 2-64424509466' 2024-06-27T21:47:13.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:47:13.950 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-27T21:47:14.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345941 2024-06-27T21:47:14.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345941 2024-06-27T21:47:14.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672997 2-64424509466 3-85899345941' 2024-06-27T21:47:14.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:47:14.092 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672997 2024-06-27T21:47:14.092 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:47:14.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T21:47:14.093 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672997 2024-06-27T21:47:14.094 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:47:14.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672997 2024-06-27T21:47:14.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2024-06-27T21:47:14.095 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672997 2024-06-27T21:47:14.095 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T21:47:14.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672995 -lt 42949672997 2024-06-27T21:47:14.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T21:47:15.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T21:47:15.518 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T21:47:15.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672997 -lt 42949672997 2024-06-27T21:47:15.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:47:15.958 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:47:15.959 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509466 2024-06-27T21:47:15.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T21:47:15.960 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509466 2024-06-27T21:47:15.960 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:47:15.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509466 2024-06-27T21:47:15.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-06-27T21:47:15.962 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509466 2024-06-27T21:47:15.962 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T21:47:16.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509466 -lt 64424509466 2024-06-27T21:47:16.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:47:16.390 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345941 2024-06-27T21:47:16.390 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:47:16.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-06-27T21:47:16.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345941 2024-06-27T21:47:16.393 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:47:16.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345941 2024-06-27T21:47:16.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-06-27T21:47:16.394 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.3 seq 85899345941 2024-06-27T21:47:16.394 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-27T21:47:16.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345941 -lt 85899345941 2024-06-27T21:47:16.828 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-06-27T21:47:16.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:47:16.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:47:17.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 == 0 2024-06-27T21:47:17.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-06-27T21:47:17.418 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-06-27T21:47:17.418 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-06-27T21:47:17.418 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-06-27T21:47:17.418 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-06-27T21:47:17.419 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-06-27T21:47:17.419 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-06-27T21:47:17.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=32 2024-06-27T21:47:17.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-06-27T21:47:17.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:47:17.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:47:18.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 32 = 32 2024-06-27T21:47:18.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-06-27T21:47:18.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-06-27T21:47:18.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-06-27T21:47:18.820 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-06-27T21:47:18.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-06-27T21:47:19.237 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:19.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-06-27T21:47:19.557 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:19.557 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-27T21:47:19.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-06-27T21:47:19.875 INFO:tasks.workunit.client.0.smithi042.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-06-27T21:47:19.876 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-06-27T21:47:19.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-27T21:47:19.881 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:47:19.881 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:47:19.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:47:19.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:47:19.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:47:20.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:47:20.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-27T21:47:20.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:47:20.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:47:20.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:47:20.194 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:47:20.194 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:47:20.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:47:20.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:47:20.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:47:20.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:47:20.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:47:20.204 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:47:20.204 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:47:20.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:47:20.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:47:20.206 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:47:20.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:20.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:47:20.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:47:20.207 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:20.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:47:20.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:47:20.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:47:20.233 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:47:20.234 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:20.234 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:20.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:47:20.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:47:20.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:47:20.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:47:20.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:47:20.237 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:47:20.238 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:47:20.238 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:47:20.239 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:47:20.239 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:47:20.239 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:47:20.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:20.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:47:20.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:47:20.241 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:20.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:47:20.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:47:20.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:47:20.244 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:47:20.244 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:20.244 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:20.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:47:20.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:47:20.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:47:20.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-27T21:47:20.246 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:47:20.246 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:20.246 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:20.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47167 2024-06-27T21:47:20.248 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:47:20.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:47:20.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:47:20.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-27T21:47:20.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-06-27T21:47:20.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-06-27T21:47:20.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-06-27T21:47:20.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-27T21:47:20.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:47:20.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:47:20.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:47:20.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:47:20.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-27T21:47:20.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-27T21:47:20.753 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:47:20.754 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:47:20.754 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:47:20.755 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:47:20.755 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:20.755 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:20.755 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:47:20.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-27T21:47:20.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:47:20.967 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:47:20.967 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:20.967 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:47:20.967 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:47:20.968 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:47:20.969 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:47:20.969 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:47:20.969 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:47:20.969 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:20.969 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:20.969 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:20.970 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:47:20.970 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:20.970 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get fsid 2024-06-27T21:47:21.090 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:47:21.090 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:21.090 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:47:21.090 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:47:21.091 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:47:21.091 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:47:21.091 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:47:21.091 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:47:21.091 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:21.091 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:21.092 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:21.092 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:47:21.092 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:21.092 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get mon_host 2024-06-27T21:47:21.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-06-27T21:47:22.035 INFO:tasks.workunit.client.0.smithi042.stderr:invalid mon(s) specified: [dne] 2024-06-27T21:47:22.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-06-27T21:47:22.235 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2024-06-27T21:47:22.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-06-27T21:47:22.455 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-06-27T21:47:22.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-06-27T21:47:22.672 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: removing mon.a would break quorum 2024-06-27T21:47:22.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-06-27T21:47:22.932 INFO:tasks.workunit.client.0.smithi042.stderr:mon.dne does not exist 2024-06-27T21:47:22.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-27T21:47:22.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:47:22.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:47:22.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:47:22.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:47:22.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:47:23.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:47:23.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-27T21:47:23.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:47:23.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:47:23.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:47:23.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:47:23.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:47:23.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:47:23.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:47:23.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:47:23.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:47:23.064 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:47:23.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:47:23.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:47:23.067 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:47:23.067 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:47:23.067 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:47:23.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:23.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:47:23.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:47:23.069 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:23.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:47:23.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:47:23.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:47:23.073 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:47:23.074 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:23.074 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:23.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:47:23.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:47:23.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:47:23.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:47:23.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:47:23.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:47:23.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:47:23.080 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:47:23.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:47:23.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:47:23.082 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:47:23.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:23.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:47:23.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:47:23.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:23.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:47:23.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:47:23.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:47:23.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:47:23.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:23.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:23.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:47:23.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:47:23.089 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:47:23.089 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-27T21:47:23.090 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:47:23.090 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:23.091 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:23.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47167 2024-06-27T21:47:23.092 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:47:23.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:47:23.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:47:23.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-27T21:47:23.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-06-27T21:47:23.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-06-27T21:47:23.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-06-27T21:47:23.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-27T21:47:23.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:47:23.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:47:23.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:47:23.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:47:23.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-27T21:47:23.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-27T21:47:23.564 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:47:23.564 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:47:23.564 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:47:23.565 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:47:23.565 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:23.565 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:23.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:47:23.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-27T21:47:23.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:47:23.785 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:47:23.785 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:23.785 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:47:23.785 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:47:23.786 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:47:23.786 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:47:23.787 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:47:23.787 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:47:23.787 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:23.787 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:23.787 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:23.787 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:47:23.787 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:23.787 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get fsid 2024-06-27T21:47:23.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:47:23.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:23.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:47:23.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:47:23.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:47:23.906 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:47:23.906 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:47:23.906 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:47:23.906 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:23.907 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:23.907 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:23.907 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:47:23.907 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:23.907 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get mon_host 2024-06-27T21:47:24.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-06-27T21:47:24.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:47:24.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:47:24.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-27T21:47:24.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:47:24.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-06-27T21:47:24.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-06-27T21:47:24.509 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:47:24.509 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:47:24.509 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:47:24.510 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:47:24.510 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:24.510 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:24.510 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:47:24.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-06-27T21:47:24.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:47:24.722 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-27T21:47:24.722 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:24.722 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T21:47:24.722 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:47:24.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:47:24.723 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T21:47:24.723 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T21:47:24.724 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T21:47:24.724 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:24.724 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:24.724 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:24.724 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.b.asok 2024-06-27T21:47:24.724 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:24.724 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.b.asok config get fsid 2024-06-27T21:47:24.841 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-27T21:47:24.841 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:24.841 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T21:47:24.841 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:47:24.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:47:24.843 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T21:47:24.843 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T21:47:24.843 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T21:47:24.843 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:24.843 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:24.843 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:24.843 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.b.asok 2024-06-27T21:47:24.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:24.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.b.asok config get mon_host 2024-06-27T21:47:24.953 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-06-27T21:47:31.190 INFO:tasks.workunit.client.0.smithi042.stderr:invalid mon(s) specified: [dne] 2024-06-27T21:47:31.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-06-27T21:47:31.474 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2024-06-27T21:47:31.477 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-06-27T21:47:31.698 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2024-06-27T21:47:31.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-06-27T21:47:31.992 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2024-06-27T21:47:31.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-06-27T21:47:32.275 INFO:tasks.workunit.client.0.smithi042.stderr:adding another mon that is not yet online will not break quorum 2024-06-27T21:47:32.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-06-27T21:47:32.652 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.a 2024-06-27T21:47:32.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-06-27T21:47:33.024 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.b 2024-06-27T21:47:33.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-06-27T21:47:33.390 INFO:tasks.workunit.client.0.smithi042.stderr:mon.dne does not exist 2024-06-27T21:47:33.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-27T21:47:33.410 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:47:33.410 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:47:33.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:47:33.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:47:33.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:47:33.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:47:33.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-27T21:47:33.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:47:33.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:47:33.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:47:33.520 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:47:33.520 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:47:33.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:47:33.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:47:33.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:47:33.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:47:33.527 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:47:33.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:47:33.528 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:47:33.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:47:33.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:47:33.530 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:47:33.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:33.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:47:33.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:47:33.532 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:33.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:47:33.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:47:33.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:47:33.537 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:47:33.537 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:33.537 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:33.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:47:33.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:47:33.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:47:33.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:47:33.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:47:33.542 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:47:33.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:47:33.543 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:47:33.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:47:33.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:47:33.545 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:47:33.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:33.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:47:33.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:47:33.547 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:47:33.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:47:33.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:47:33.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:47:33.550 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:47:33.550 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:33.551 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:33.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:47:33.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:47:33.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:47:33.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-27T21:47:33.553 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:47:33.554 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:33.554 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:33.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47167 2024-06-27T21:47:33.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:47:33.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:47:33.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:47:33.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-27T21:47:33.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-06-27T21:47:33.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-06-27T21:47:33.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-06-27T21:47:33.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-27T21:47:33.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:47:33.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:47:33.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:47:33.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:47:33.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-27T21:47:33.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-27T21:47:34.019 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:47:34.019 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:47:34.019 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:47:34.019 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:47:34.019 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:34.019 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:34.020 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:47:34.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-27T21:47:34.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:47:34.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:47:34.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:34.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:47:34.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:47:34.250 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:47:34.250 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:47:34.250 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:47:34.250 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:47:34.251 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:34.251 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:34.251 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:34.251 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:47:34.251 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:34.251 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get fsid 2024-06-27T21:47:34.369 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:47:34.370 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:34.370 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:47:34.370 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:47:34.371 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:47:34.371 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:47:34.371 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:47:34.371 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:47:34.371 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:34.371 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:34.371 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:34.371 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:47:34.372 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:34.372 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get mon_host 2024-06-27T21:47:34.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-06-27T21:47:34.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:47:34.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:47:34.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-27T21:47:34.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:47:34.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-06-27T21:47:34.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-06-27T21:47:34.980 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:47:34.980 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:47:34.980 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:47:34.981 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:47:34.981 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:34.981 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:34.981 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:47:34.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-06-27T21:47:35.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:47:35.218 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-27T21:47:35.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:35.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T21:47:35.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:47:35.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:47:35.220 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T21:47:35.220 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T21:47:35.220 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T21:47:35.220 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:35.220 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:35.220 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:35.220 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.b.asok 2024-06-27T21:47:35.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:35.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.b.asok config get fsid 2024-06-27T21:47:35.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-27T21:47:35.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:35.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T21:47:35.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:47:35.339 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:47:35.339 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T21:47:35.339 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T21:47:35.339 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T21:47:35.339 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:35.339 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:35.340 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:35.340 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.b.asok 2024-06-27T21:47:35.340 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:35.340 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.b.asok config get mon_host 2024-06-27T21:47:35.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-06-27T21:47:35.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:47:35.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:47:35.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-06-27T21:47:35.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:47:35.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-06-27T21:47:35.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-06-27T21:47:35.968 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:47:35.968 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:47:35.968 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:47:35.969 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:47:35.969 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:35.969 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:35.969 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:47:35.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-06-27T21:47:36.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:47:36.196 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-06-27T21:47:36.196 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:36.196 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-27T21:47:36.197 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:47:36.197 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:47:36.198 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-27T21:47:36.198 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-27T21:47:36.198 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-27T21:47:36.198 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:36.199 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:36.199 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:36.199 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.c.asok 2024-06-27T21:47:36.199 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:36.199 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.c.asok config get fsid 2024-06-27T21:47:36.316 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-06-27T21:47:36.316 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:47:36.316 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-27T21:47:36.317 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:47:36.318 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:47:36.318 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-27T21:47:36.318 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-27T21:47:36.318 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-27T21:47:36.318 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:47:36.318 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:47:36.318 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:47:36.318 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.c.asok 2024-06-27T21:47:36.318 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:47:36.319 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.c.asok config get mon_host 2024-06-27T21:47:36.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-06-27T21:47:36.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-27T21:47:36.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-06-27T21:47:36.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-27T21:47:36.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-06-27T21:47:36.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-27T21:47:36.432 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-27T21:47:36.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1719524916 2024-06-27T21:47:36.432 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:36.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524856 -lt 1719524916 ]] 2024-06-27T21:47:36.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:47:36.433 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:39.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:39.680 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:39.680 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:47:39.681 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:39.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-27T21:47:39.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-27T21:47:39.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-27T21:47:39.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-27T21:47:39.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-27T21:47:39.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-06-27T21:47:39.982 INFO:tasks.workunit.client.0.smithi042.stderr:invalid mon(s) specified: [dne] 2024-06-27T21:47:39.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-06-27T21:47:40.351 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (b,c) after stopping [a] 2024-06-27T21:47:40.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-06-27T21:47:43.593 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (a,c) after stopping [b] 2024-06-27T21:47:43.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-06-27T21:47:43.922 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (a,b) after stopping [c] 2024-06-27T21:47:43.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-06-27T21:47:44.224 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2024-06-27T21:47:44.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-06-27T21:47:44.485 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2024-06-27T21:47:44.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-06-27T21:47:44.728 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2024-06-27T21:47:44.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-06-27T21:47:45.023 INFO:tasks.workunit.client.0.smithi042.stderr:adding another mon that is not yet online will not break quorum 2024-06-27T21:47:45.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-06-27T21:47:45.395 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.a 2024-06-27T21:47:45.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-06-27T21:47:45.772 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.b 2024-06-27T21:47:45.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-06-27T21:47:46.121 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.c 2024-06-27T21:47:46.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-06-27T21:47:46.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:47:46.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:47:46.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:47:46.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:47:46.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:47:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:47:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-06-27T21:47:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-27T21:47:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2024-06-27T21:47:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-27T21:47:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2024-06-27T21:47:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-27T21:47:46.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-27T21:47:46.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1719524926 2024-06-27T21:47:46.250 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:46.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524866 -lt 1719524926 ]] 2024-06-27T21:47:46.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:46.251 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:46.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:46.495 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:46.495 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:47:46.496 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:46.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:47:46.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:47:46.526 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:46.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524866 -lt 1719524926 ]] 2024-06-27T21:47:46.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:46.527 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:49.776 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:49.776 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:49.777 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:47:49.777 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:49.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:47:49.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:47:49.814 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:49.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524869 -lt 1719524926 ]] 2024-06-27T21:47:49.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:49.815 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:50.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:50.065 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:50.065 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:50.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:47:50.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:47:50.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:47:50.094 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:50.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524870 -lt 1719524926 ]] 2024-06-27T21:47:50.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:50.095 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:50.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:50.317 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:50.318 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:50.318 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:47:50.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:47:50.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:47:50.351 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:50.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524870 -lt 1719524926 ]] 2024-06-27T21:47:50.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:50.352 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:50.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:50.591 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:50.591 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:50.591 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:47:50.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:47:50.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:47:50.628 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:50.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524870 -lt 1719524926 ]] 2024-06-27T21:47:50.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:50.629 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:50.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:50.858 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:50.859 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:47:50.859 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:50.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:47:50.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:47:50.895 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:50.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524870 -lt 1719524926 ]] 2024-06-27T21:47:50.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:50.897 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:51.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:51.126 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:51.126 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:51.126 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:47:51.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:47:51.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:47:51.164 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:51.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524871 -lt 1719524926 ]] 2024-06-27T21:47:51.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:51.165 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:54.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:54.403 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:54.404 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:54.404 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:47:54.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:47:54.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:47:54.440 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:54.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524874 -lt 1719524926 ]] 2024-06-27T21:47:54.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:54.441 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:54.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:54.681 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:54.681 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:47:54.681 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:54.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:47:54.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:47:54.717 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:54.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524874 -lt 1719524926 ]] 2024-06-27T21:47:54.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:54.718 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:54.951 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:54.952 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:54.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:47:54.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:54.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:47:54.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:47:54.987 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:54.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524874 -lt 1719524926 ]] 2024-06-27T21:47:54.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:54.988 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:47:55.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:47:55.214 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:55.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:47:55.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:47:55.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:47:55.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:47:55.244 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:47:55.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524875 -lt 1719524926 ]] 2024-06-27T21:47:55.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-06-27T21:47:55.246 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:01.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:01.665 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:01.666 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-06-27T21:48:01.666 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:47:33.579987Z","created":"2024-06-27T21:47:33.579987Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:01.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-27T21:48:01.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-27T21:48:01.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-27T21:48:01.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-27T21:48:01.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-27T21:48:01.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-06-27T21:48:01.982 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2024-06-27T21:48:01.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-06-27T21:48:05.224 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (a,c) after stopping [b] 2024-06-27T21:48:05.239 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-06-27T21:48:05.504 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2024-06-27T21:48:05.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-06-27T21:48:05.759 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-06-27T21:48:05.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-06-27T21:48:05.951 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: removing mon.a would break quorum 2024-06-27T21:48:05.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-06-27T21:48:06.245 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.b 2024-06-27T21:48:06.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-06-27T21:48:06.533 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: removing mon.c would break quorum 2024-06-27T21:48:06.536 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-27T21:48:06.537 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:48:06.538 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:48:06.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:48:06.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:48:06.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:48:06.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:48:06.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-27T21:48:06.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:48:06.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:48:06.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:48:06.646 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:48:06.646 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:48:06.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:48:06.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:48:06.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:48:06.651 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:48:06.652 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:48:06.652 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:48:06.653 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:48:06.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:48:06.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:48:06.654 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:48:06.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:48:06.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:48:06.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:48:06.656 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:48:06.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:48:06.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:48:06.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:48:06.663 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:48:06.663 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:06.663 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:06.663 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:48:06.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:48:06.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:48:06.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:48:06.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:48:06.666 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:48:06.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:48:06.668 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:48:06.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:48:06.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:48:06.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:48:06.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:48:06.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:48:06.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:48:06.670 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:48:06.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:48:06.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:48:06.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:48:06.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:48:06.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:06.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:06.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:48:06.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:48:06.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:48:06.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-27T21:48:06.676 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:48:06.676 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:06.677 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:06.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47167 2024-06-27T21:48:06.679 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:48:06.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:48:06.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:48:06.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-27T21:48:06.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-06-27T21:48:06.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-06-27T21:48:06.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-06-27T21:48:06.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-27T21:48:06.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:48:06.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:48:06.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:48:06.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:48:06.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-27T21:48:06.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-27T21:48:07.170 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:48:07.186 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:48:07.186 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:48:07.186 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:48:07.186 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:07.186 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:07.186 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:48:07.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-27T21:48:07.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:48:07.378 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:48:07.378 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:07.379 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:48:07.379 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:48:07.379 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:48:07.379 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:48:07.379 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:48:07.379 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:48:07.380 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:07.380 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:07.380 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:07.380 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:48:07.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:07.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get fsid 2024-06-27T21:48:07.498 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:48:07.498 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:07.498 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:48:07.498 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:48:07.498 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:48:07.498 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:48:07.498 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:48:07.498 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:48:07.498 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:07.498 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:07.499 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:07.499 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:48:07.499 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:07.499 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get mon_host 2024-06-27T21:48:07.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-06-27T21:48:07.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:48:07.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:48:07.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-27T21:48:07.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:48:07.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-06-27T21:48:07.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-06-27T21:48:08.126 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:48:08.126 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:48:08.127 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:48:08.127 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:48:08.127 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:08.128 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:08.128 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:48:08.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-06-27T21:48:08.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:48:08.346 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-27T21:48:08.346 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:08.346 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T21:48:08.346 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:48:08.347 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:48:08.347 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T21:48:08.347 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T21:48:08.347 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T21:48:08.348 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:08.348 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:08.348 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:08.348 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.b.asok 2024-06-27T21:48:08.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:08.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.b.asok config get fsid 2024-06-27T21:48:08.464 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-27T21:48:08.464 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:08.465 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T21:48:08.465 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:48:08.466 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T21:48:08.466 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:48:08.466 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T21:48:08.466 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T21:48:08.466 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:08.466 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:08.466 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:08.466 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.b.asok 2024-06-27T21:48:08.466 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:08.466 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.b.asok config get mon_host 2024-06-27T21:48:08.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-06-27T21:48:08.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:48:08.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:48:08.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-06-27T21:48:08.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:48:08.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-06-27T21:48:08.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-06-27T21:48:09.097 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:48:09.098 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:48:09.098 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:48:09.098 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:48:09.098 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:09.099 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:09.099 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:48:09.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-06-27T21:48:09.318 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:48:09.319 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-06-27T21:48:09.319 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:09.319 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-27T21:48:09.319 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:48:09.320 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:48:09.320 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-27T21:48:09.320 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-27T21:48:09.320 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-27T21:48:09.320 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:09.321 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:09.321 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:09.321 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.c.asok 2024-06-27T21:48:09.321 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:09.321 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.c.asok config get fsid 2024-06-27T21:48:09.441 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-06-27T21:48:09.441 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:09.441 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-27T21:48:09.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:48:09.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:48:09.442 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-27T21:48:09.442 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-27T21:48:09.442 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-27T21:48:09.443 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:09.443 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:09.443 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:09.443 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.c.asok 2024-06-27T21:48:09.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:09.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.c.asok config get mon_host 2024-06-27T21:48:09.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-06-27T21:48:09.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:48:09.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:48:09.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-06-27T21:48:09.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:48:09.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-06-27T21:48:09.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-06-27T21:48:10.077 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:48:10.077 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:48:10.077 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:48:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:48:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:10.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:48:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-06-27T21:48:10.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:48:10.289 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-06-27T21:48:10.289 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:10.289 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-06-27T21:48:10.289 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:48:10.290 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:48:10.290 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-06-27T21:48:10.290 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-06-27T21:48:10.290 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-06-27T21:48:10.290 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:10.290 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:10.290 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:10.290 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.d.asok 2024-06-27T21:48:10.291 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:10.291 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.d.asok config get fsid 2024-06-27T21:48:10.410 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-06-27T21:48:10.410 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:10.410 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-06-27T21:48:10.410 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:48:10.411 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:48:10.411 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-06-27T21:48:10.411 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-06-27T21:48:10.411 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-06-27T21:48:10.412 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:10.412 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:10.412 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:10.412 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.d.asok 2024-06-27T21:48:10.412 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:10.412 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.d.asok config get mon_host 2024-06-27T21:48:10.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-06-27T21:48:10.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-27T21:48:10.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-06-27T21:48:10.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-27T21:48:10.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-06-27T21:48:10.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-27T21:48:10.526 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-27T21:48:10.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1719524950 2024-06-27T21:48:10.527 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:10.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524890 -lt 1719524950 ]] 2024-06-27T21:48:10.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:48:10.529 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:22.774 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:22.774 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:22.774 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:22.775 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:48:22.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-27T21:48:22.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-27T21:48:22.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-27T21:48:22.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-27T21:48:22.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-27T21:48:22.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-06-27T21:48:23.135 INFO:tasks.workunit.client.0.smithi042.stderr:invalid mon(s) specified: [dne] 2024-06-27T21:48:23.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-06-27T21:48:23.500 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-06-27T21:48:23.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-06-27T21:48:23.846 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (a,c,d) after stopping [b] 2024-06-27T21:48:23.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-06-27T21:48:24.234 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (a,b,d) after stopping [c] 2024-06-27T21:48:24.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-06-27T21:48:24.566 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (a,b,c) after stopping [d] 2024-06-27T21:48:24.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-06-27T21:48:24.883 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2024-06-27T21:48:24.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-06-27T21:48:25.204 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2024-06-27T21:48:25.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-06-27T21:48:25.565 INFO:tasks.workunit.client.0.smithi042.stderr:adding another mon that is not yet online will not break quorum 2024-06-27T21:48:25.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-06-27T21:48:25.885 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.a 2024-06-27T21:48:25.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-06-27T21:48:26.230 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.b 2024-06-27T21:48:26.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-06-27T21:48:26.616 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.c 2024-06-27T21:48:26.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-06-27T21:48:26.633 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:48:26.633 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:48:26.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:48:26.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:48:26.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:48:26.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:48:26.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-06-27T21:48:26.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-27T21:48:26.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-06-27T21:48:26.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-27T21:48:26.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-06-27T21:48:26.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-27T21:48:26.743 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-27T21:48:26.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1719524966 2024-06-27T21:48:26.744 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:26.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524906 -lt 1719524966 ]] 2024-06-27T21:48:26.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:26.745 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:26.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:26.986 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:26.987 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:26.987 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:27.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:48:27.023 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:48:27.023 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:27.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524907 -lt 1719524966 ]] 2024-06-27T21:48:27.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:27.025 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:27.264 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:27.265 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:27.265 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:27.266 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:27.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:48:27.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:48:27.302 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:27.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524907 -lt 1719524966 ]] 2024-06-27T21:48:27.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:27.303 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:27.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:27.541 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:27.542 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:27.542 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:27.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:48:27.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:48:27.578 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:27.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524907 -lt 1719524966 ]] 2024-06-27T21:48:27.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:27.579 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:27.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:27.822 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:27.822 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:27.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:27.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:48:27.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:48:27.859 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:27.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524907 -lt 1719524966 ]] 2024-06-27T21:48:27.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:27.860 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:28.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:28.091 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:28.092 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:28.092 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:28.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:48:28.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:48:28.122 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:28.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524908 -lt 1719524966 ]] 2024-06-27T21:48:28.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:28.123 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:28.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:28.364 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:28.364 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:28.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:28.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:48:28.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:48:28.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:28.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524908 -lt 1719524966 ]] 2024-06-27T21:48:28.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:28.402 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:28.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:28.643 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:28.644 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:28.644 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:28.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:48:28.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:48:28.680 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:28.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524908 -lt 1719524966 ]] 2024-06-27T21:48:28.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:28.681 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:31.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:31.924 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:31.925 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:31.926 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:31.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:48:31.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:48:31.961 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:31.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524911 -lt 1719524966 ]] 2024-06-27T21:48:31.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:31.962 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:35.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:35.214 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:35.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:35.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:35.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:48:35.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:48:35.241 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:35.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524915 -lt 1719524966 ]] 2024-06-27T21:48:35.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:35.242 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:35.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:35.483 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:35.483 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:35.483 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:35.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:48:35.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:48:35.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:35.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524915 -lt 1719524966 ]] 2024-06-27T21:48:35.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:35.526 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:35.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:35.760 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:35.760 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:35.761 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:35.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:48:35.795 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:48:35.795 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:48:35.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524915 -lt 1719524966 ]] 2024-06-27T21:48:35.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:48:35.797 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:48:48.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:48:48.036 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:48.036 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:06.698237Z","created":"2024-06-27T21:48:06.698237Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:48:48.037 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:48:48.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-27T21:48:48.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-27T21:48:48.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-27T21:48:48.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-27T21:48:48.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-27T21:48:48.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-06-27T21:48:48.352 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-06-27T21:48:48.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-06-27T21:48:48.654 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-06-27T21:48:48.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-06-27T21:48:48.935 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-06-27T21:48:48.939 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-06-27T21:48:49.178 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-06-27T21:48:49.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-06-27T21:48:49.503 INFO:tasks.workunit.client.0.smithi042.stderr:adding another mon that is not yet online will not break quorum 2024-06-27T21:48:49.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-06-27T21:48:52.751 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.a 2024-06-27T21:48:52.767 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-06-27T21:48:56.000 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.b 2024-06-27T21:48:56.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-06-27T21:48:56.348 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.c 2024-06-27T21:48:56.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-06-27T21:48:56.693 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.d 2024-06-27T21:48:56.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-27T21:48:56.709 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:48:56.710 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:48:56.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:48:56.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:48:56.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:48:56.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:48:56.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-27T21:48:56.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:48:56.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:48:56.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:48:56.822 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:48:56.822 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:48:56.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:48:56.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:48:56.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:48:56.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:48:56.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:48:56.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:48:56.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:48:56.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:48:56.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:48:56.831 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:48:56.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:48:56.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:48:56.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:48:56.832 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:48:56.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:48:56.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:48:56.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:48:56.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:48:56.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:56.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:56.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:48:56.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:48:56.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:48:56.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-06-27T21:48:56.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-06-27T21:48:56.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-06-27T21:48:56.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-06-27T21:48:56.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:48:56.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:48:56.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:48:56.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:48:56.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:48:56.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:48:56.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:48:56.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:48:56.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:48:56.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:48:56.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:48:56.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:48:56.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:48:56.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:48:56.847 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:48:56.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:48:56.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:48:56.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:48:56.849 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:48:56.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:48:56.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:48:56.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:48:56.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:48:56.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:56.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:56.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:48:56.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:48:56.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:48:56.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-06-27T21:48:56.853 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:48:56.853 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:56.853 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:56.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47167 2024-06-27T21:48:56.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:48:56.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:48:56.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:48:56.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-06-27T21:48:56.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-06-27T21:48:56.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-06-27T21:48:56.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=5e63fabe-88bf-4ba0-9243-93aaae2581f0 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-06-27T21:48:56.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-06-27T21:48:56.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:48:56.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:48:56.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:48:56.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:48:56.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-06-27T21:48:56.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-06-27T21:48:57.327 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:48:57.327 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:48:57.328 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:48:57.328 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:48:57.328 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:57.328 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:57.329 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:48:57.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-06-27T21:48:57.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:48:57.600 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:48:57.600 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:57.600 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:48:57.600 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:48:57.601 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:48:57.601 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:48:57.601 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:48:57.601 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:48:57.602 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:57.602 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:57.602 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:57.602 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:48:57.602 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:57.602 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get fsid 2024-06-27T21:48:57.721 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:48:57.721 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:57.721 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:48:57.721 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:48:57.722 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:48:57.722 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:48:57.722 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:48:57.723 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:48:57.723 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:57.723 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:57.723 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:57.723 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.a.asok 2024-06-27T21:48:57.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:57.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.a.asok config get mon_host 2024-06-27T21:48:57.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-06-27T21:48:57.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:48:57.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:48:57.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-27T21:48:57.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:48:57.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-06-27T21:48:57.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-06-27T21:48:58.289 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:48:58.289 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:48:58.290 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:48:58.290 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:48:58.290 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:58.291 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:58.291 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:48:58.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-06-27T21:48:58.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:48:58.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-27T21:48:58.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:58.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T21:48:58.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:48:58.504 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:48:58.504 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T21:48:58.504 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T21:48:58.504 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T21:48:58.504 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:58.504 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:58.504 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:58.504 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.b.asok 2024-06-27T21:48:58.504 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:58.504 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.b.asok config get fsid 2024-06-27T21:48:58.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-27T21:48:58.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:58.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T21:48:58.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:48:58.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:48:58.624 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T21:48:58.624 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T21:48:58.625 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T21:48:58.625 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:58.625 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:58.625 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:58.625 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.b.asok 2024-06-27T21:48:58.625 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:58.625 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.b.asok config get mon_host 2024-06-27T21:48:58.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-06-27T21:48:58.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:48:58.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:48:58.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-06-27T21:48:58.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:48:58.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-06-27T21:48:58.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-06-27T21:48:59.252 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:48:59.252 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:48:59.252 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:48:59.253 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:48:59.253 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:59.253 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:59.253 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:48:59.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-06-27T21:48:59.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:48:59.465 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-06-27T21:48:59.465 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:59.465 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-27T21:48:59.465 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:48:59.466 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:48:59.466 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-27T21:48:59.466 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-27T21:48:59.466 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-27T21:48:59.467 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:59.467 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:59.467 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:59.467 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.c.asok 2024-06-27T21:48:59.467 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:59.467 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.c.asok config get fsid 2024-06-27T21:48:59.584 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-06-27T21:48:59.584 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:48:59.584 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-06-27T21:48:59.584 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:48:59.584 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:48:59.585 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-06-27T21:48:59.585 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-06-27T21:48:59.585 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-06-27T21:48:59.585 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:48:59.586 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:48:59.586 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:48:59.586 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.c.asok 2024-06-27T21:48:59.586 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:48:59.586 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.c.asok config get mon_host 2024-06-27T21:48:59.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-06-27T21:48:59.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:48:59.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:48:59.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-06-27T21:48:59.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:48:59.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-06-27T21:48:59.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-06-27T21:49:00.217 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:49:00.217 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:49:00.217 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:49:00.217 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:49:00.217 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:49:00.217 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:49:00.217 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:49:00.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-06-27T21:49:00.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:49:00.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-06-27T21:49:00.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:49:00.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-06-27T21:49:00.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:49:00.438 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:49:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-06-27T21:49:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-06-27T21:49:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-06-27T21:49:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:49:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:49:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:49:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.d.asok 2024-06-27T21:49:00.439 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:49:00.439 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.d.asok config get fsid 2024-06-27T21:49:00.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-06-27T21:49:00.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:49:00.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-06-27T21:49:00.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:49:00.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:49:00.557 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-06-27T21:49:00.557 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-06-27T21:49:00.557 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-06-27T21:49:00.557 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:49:00.558 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:49:00.558 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:49:00.558 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.d.asok 2024-06-27T21:49:00.558 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:49:00.558 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.d.asok config get mon_host 2024-06-27T21:49:00.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-06-27T21:49:00.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-06-27T21:49:00.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:49:00.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2024-06-27T21:49:00.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:49:00.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2024-06-27T21:49:00.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-06-27T21:49:01.229 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:49:01.229 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:49:01.229 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:49:01.229 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:49:01.230 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:49:01.230 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:49:01.230 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47167/$cluster-$name.asok' 2024-06-27T21:49:01.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47167/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-06-27T21:49:01.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:49:01.458 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2024-06-27T21:49:01.458 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:49:01.458 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-06-27T21:49:01.458 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:49:01.459 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:49:01.459 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-06-27T21:49:01.459 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-06-27T21:49:01.459 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-06-27T21:49:01.459 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:49:01.459 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:49:01.460 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:49:01.460 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.e.asok 2024-06-27T21:49:01.460 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:49:01.460 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.e.asok config get fsid 2024-06-27T21:49:01.581 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2024-06-27T21:49:01.581 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:49:01.581 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-06-27T21:49:01.581 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:49:01.582 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:49:01.583 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-06-27T21:49:01.583 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-06-27T21:49:01.583 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-06-27T21:49:01.583 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:49:01.583 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:49:01.583 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:49:01.583 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47167/ceph-mon.e.asok 2024-06-27T21:49:01.583 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:49:01.584 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47167/ceph-mon.e.asok config get mon_host 2024-06-27T21:49:01.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-06-27T21:49:01.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-27T21:49:01.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2024-06-27T21:49:01.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-27T21:49:01.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2024-06-27T21:49:01.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-27T21:49:01.698 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-27T21:49:01.699 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1719525001 2024-06-27T21:49:01.699 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:01.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524941 -lt 1719525001 ]] 2024-06-27T21:49:01.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-06-27T21:49:01.701 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:06.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:06.359 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:06.359 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2024-06-27T21:49:06.360 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:06.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-27T21:49:06.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-27T21:49:06.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-27T21:49:06.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-27T21:49:06.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-27T21:49:06.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-06-27T21:49:06.753 INFO:tasks.workunit.client.0.smithi042.stderr:invalid mon(s) specified: [dne] 2024-06-27T21:49:06.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-06-27T21:49:07.097 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-06-27T21:49:07.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-06-27T21:49:07.582 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2024-06-27T21:49:07.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-06-27T21:49:10.833 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2024-06-27T21:49:10.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-06-27T21:49:11.205 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2024-06-27T21:49:11.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-06-27T21:49:11.601 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2024-06-27T21:49:11.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-06-27T21:49:11.923 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2024-06-27T21:49:11.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-06-27T21:49:12.350 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2024-06-27T21:49:12.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-06-27T21:49:15.562 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2024-06-27T21:49:15.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-06-27T21:49:15.929 INFO:tasks.workunit.client.0.smithi042.stderr:adding another mon that is not yet online will not break quorum 2024-06-27T21:49:15.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-06-27T21:49:16.498 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.a 2024-06-27T21:49:16.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-06-27T21:49:16.934 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.b 2024-06-27T21:49:16.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-06-27T21:49:17.233 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.c 2024-06-27T21:49:17.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-06-27T21:49:17.574 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.d 2024-06-27T21:49:17.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-06-27T21:49:18.159 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.e 2024-06-27T21:49:18.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-06-27T21:49:18.182 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:49:18.182 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:49:18.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:49:18.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:49:18.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:49:18.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:49:18.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-06-27T21:49:18.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-27T21:49:18.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-06-27T21:49:18.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-27T21:49:18.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-06-27T21:49:18.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-27T21:49:18.292 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-27T21:49:18.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1719525018 2024-06-27T21:49:18.294 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:18.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524958 -lt 1719525018 ]] 2024-06-27T21:49:18.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:18.296 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:18.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:18.538 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:18.540 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:18.540 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:18.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:18.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:18.576 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:18.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524958 -lt 1719525018 ]] 2024-06-27T21:49:18.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:18.577 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:24.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:24.823 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:24.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:24.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:24.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:24.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:24.860 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:24.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524964 -lt 1719525018 ]] 2024-06-27T21:49:24.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:24.861 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:25.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:25.099 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:25.099 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:25.099 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:25.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:25.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:25.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:25.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524965 -lt 1719525018 ]] 2024-06-27T21:49:25.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:25.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:25.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:25.365 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:25.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:25.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:25.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:25.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:25.394 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:25.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524965 -lt 1719525018 ]] 2024-06-27T21:49:25.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:25.395 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:25.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:25.628 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:25.628 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:25.628 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:25.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:25.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:25.664 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:25.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524965 -lt 1719525018 ]] 2024-06-27T21:49:25.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:25.665 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:25.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:25.895 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:25.895 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:25.896 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:25.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:25.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:25.922 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:25.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524965 -lt 1719525018 ]] 2024-06-27T21:49:25.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:25.923 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:26.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:26.150 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:26.151 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:26.151 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:26.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:26.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:26.184 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:26.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524966 -lt 1719525018 ]] 2024-06-27T21:49:26.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:26.185 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:26.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:26.416 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:26.416 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:26.416 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:26.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:26.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:26.446 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:26.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524966 -lt 1719525018 ]] 2024-06-27T21:49:26.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:26.448 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:26.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:26.688 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:26.688 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:26.688 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:26.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:26.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:26.724 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:26.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524966 -lt 1719525018 ]] 2024-06-27T21:49:26.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:26.726 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:26.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:26.954 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:26.955 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:26.955 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:26.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:26.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:26.990 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:26.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524966 -lt 1719525018 ]] 2024-06-27T21:49:26.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:26.991 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:27.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:27.230 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:27.230 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:27.230 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:27.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:27.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:27.267 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:27.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524967 -lt 1719525018 ]] 2024-06-27T21:49:27.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:27.268 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:27.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:27.504 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:27.505 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:27.505 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:27.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:27.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:27.535 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:27.536 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524967 -lt 1719525018 ]] 2024-06-27T21:49:27.536 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:27.536 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:27.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:27.771 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:27.772 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:27.772 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:27.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:27.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:27.806 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:27.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524967 -lt 1719525018 ]] 2024-06-27T21:49:27.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-06-27T21:49:27.807 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:34.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:34.186 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:34.187 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:34.187 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-06-27T21:49:34.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-27T21:49:34.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-27T21:49:34.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-27T21:49:34.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-27T21:49:34.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-27T21:49:34.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-06-27T21:49:34.599 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-06-27T21:49:34.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-06-27T21:49:34.978 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (c,d,e) after stopping [b] 2024-06-27T21:49:34.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-06-27T21:49:38.255 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (b,d,e) after stopping [c] 2024-06-27T21:49:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-06-27T21:49:41.504 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (b,c,e) after stopping [d] 2024-06-27T21:49:41.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-06-27T21:49:41.862 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-06-27T21:49:41.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-06-27T21:49:42.236 INFO:tasks.workunit.client.0.smithi042.stderr:adding another mon that is not yet online will not break quorum 2024-06-27T21:49:42.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-06-27T21:49:42.605 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.a 2024-06-27T21:49:42.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-06-27T21:49:43.001 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.b 2024-06-27T21:49:43.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-06-27T21:49:43.344 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.c 2024-06-27T21:49:43.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-06-27T21:49:43.718 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.d 2024-06-27T21:49:43.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-06-27T21:49:44.083 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.e 2024-06-27T21:49:44.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-06-27T21:49:44.099 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:49:44.099 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:49:44.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:49:44.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:49:44.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:49:44.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:49:44.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-06-27T21:49:44.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-06-27T21:49:44.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-06-27T21:49:44.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-06-27T21:49:44.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-06-27T21:49:44.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-06-27T21:49:44.208 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-06-27T21:49:44.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1719525044 2024-06-27T21:49:44.210 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:44.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524984 -lt 1719525044 ]] 2024-06-27T21:49:44.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:44.211 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:44.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:44.447 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:44.447 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:44.447 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:44.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:44.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:44.487 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:44.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524984 -lt 1719525044 ]] 2024-06-27T21:49:44.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:44.488 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:44.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:44.712 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:44.713 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:44.713 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:44.747 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:44.747 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:44.747 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:44.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524984 -lt 1719525044 ]] 2024-06-27T21:49:44.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:44.749 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:45.002 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:45.002 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:45.002 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:45.003 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:45.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:45.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:45.031 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:45.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524985 -lt 1719525044 ]] 2024-06-27T21:49:45.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:45.032 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:48.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:48.263 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:48.264 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:48.264 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:48.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:48.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:48.302 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:48.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524988 -lt 1719525044 ]] 2024-06-27T21:49:48.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:48.304 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:51.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:51.544 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:51.544 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:51.545 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:51.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:51.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:51.573 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:51.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524991 -lt 1719525044 ]] 2024-06-27T21:49:51.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:51.574 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:51.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:51.808 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:51.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:51.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:51.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:51.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:51.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:51.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524991 -lt 1719525044 ]] 2024-06-27T21:49:51.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:51.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:52.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:52.083 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:52.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:52.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:52.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:52.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:52.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:52.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524992 -lt 1719525044 ]] 2024-06-27T21:49:52.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:52.120 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:52.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:52.357 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:52.358 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:52.358 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:52.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:52.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:52.388 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:52.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524992 -lt 1719525044 ]] 2024-06-27T21:49:52.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:52.389 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:52.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:52.617 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:52.617 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:52.618 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:52.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:52.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:52.645 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:52.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524992 -lt 1719525044 ]] 2024-06-27T21:49:52.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:52.646 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:52.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:52.883 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:52.883 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:52.884 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:52.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:52.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:52.920 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:52.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524992 -lt 1719525044 ]] 2024-06-27T21:49:52.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:52.922 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:53.156 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:53.156 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:53.156 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:53.157 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:53.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:53.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:53.192 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:53.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524993 -lt 1719525044 ]] 2024-06-27T21:49:53.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:53.193 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:53.427 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:53.427 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:53.427 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:53.428 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:53.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-06-27T21:49:53.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-06-27T21:49:53.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-06-27T21:49:53.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1719524993 -lt 1719525044 ]] 2024-06-27T21:49:53.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-06-27T21:49:53.458 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-06-27T21:49:59.892 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-06-27T21:49:59.893 INFO:tasks.workunit.client.0.smithi042.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:59.893 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-06-27T21:49:59.893 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"5e63fabe-88bf-4ba0-9243-93aaae2581f0","modified":"2024-06-27T21:48:56.875346Z","created":"2024-06-27T21:48:56.875346Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-06-27T21:49:59.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-06-27T21:49:59.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-06-27T21:49:59.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-06-27T21:49:59.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-06-27T21:49:59.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-06-27T21:49:59.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-06-27T21:50:03.166 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-06-27T21:50:03.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-06-27T21:50:03.478 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-06-27T21:50:03.481 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-06-27T21:50:03.748 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-06-27T21:50:03.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-06-27T21:50:06.940 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-06-27T21:50:06.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-06-27T21:50:10.167 INFO:tasks.workunit.client.0.smithi042.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-06-27T21:50:10.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-06-27T21:50:13.376 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-06-27T21:50:13.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-06-27T21:50:16.605 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.a 2024-06-27T21:50:16.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-06-27T21:50:16.923 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: removing mon.b would break quorum 2024-06-27T21:50:16.929 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-06-27T21:50:17.206 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: removing mon.c would break quorum 2024-06-27T21:50:17.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-06-27T21:50:17.474 INFO:tasks.workunit.client.0.smithi042.stderr:Error EBUSY: removing mon.d would break quorum 2024-06-27T21:50:17.478 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-06-27T21:50:20.713 INFO:tasks.workunit.client.0.smithi042.stderr:safe to remove mon.e 2024-06-27T21:50:20.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-06-27T21:50:20.729 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:50:20.729 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:50:20.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:50:20.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:50:20.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:50:20.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:50:20.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-06-27T21:50:20.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:50:20.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:50:20.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:50:20.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:50:20.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:50:20.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:50:20.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:50:20.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:50:20.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:50:20.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:50:20.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:50:20.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:50:20.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:50:20.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:50:20.853 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:50:20.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:50:20.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:50:20.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:50:20.853 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:50:20.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:50:20.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:50:20.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:50:20.865 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:50:20.865 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:50:20.865 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:50:20.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:50:20.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:50:20.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:50:20.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-06-27T21:50:20.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/ok-to-stop 0 2024-06-27T21:50:20.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-06-27T21:50:20.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-27T21:50:20.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-06-27T21:50:20.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:50:20.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:50:20.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:50:20.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:50:20.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:50:20.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:50:20.869 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:50:20.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:50:20.870 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:50:20.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:50:20.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:50:20.871 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:50:20.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:50:20.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:50:20.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:50:20.873 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:50:20.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:50:20.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-27T21:50:20.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-06-27T21:50:20.875 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:50:20.876 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:50:20.876 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47167 2024-06-27T21:50:20.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47167 2024-06-27T21:50:20.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:50:20.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:50:20.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-06-27T21:50:20.878 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-27T21:50:20.878 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-27T21:50:20.888 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-06-27T21:50:20.888 DEBUG:teuthology.orchestra.run.smithi042:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-06-27T21:50:20.939 INFO:tasks.workunit.client.0.smithi042.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-27T21:50:20.944 INFO:tasks.workunit.client.0.smithi042.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-27T21:50:20.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/rados-striper 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-06-27T21:50:20.945 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=10fb0861-c531-441c-adaf-3e91d91d97be --auth-supported=none ' 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:50:20.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:50:20.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:50:20.948 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:50:20.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:50:20.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:50:20.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:50:20.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:50:20.950 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:50:20.951 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:50:20.951 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:50:20.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:50:20.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:50:20.953 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:50:20.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:50:20.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-06-27T21:50:20.955 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:50:20.955 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:50:20.955 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63994 2024-06-27T21:50:20.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.63994 2024-06-27T21:50:20.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:50:20.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:50:20.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2024-06-27T21:50:20.958 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:50:20.958 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:50:20.958 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63994 2024-06-27T21:50:20.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.63994 2024-06-27T21:50:20.961 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:50:20.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-27T21:50:20.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-27T21:50:20.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:50:20.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-06-27T21:50:20.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-06-27T21:50:20.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2024-06-27T21:50:20.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:50:20.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:50:20.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:50:20.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2024-06-27T21:50:20.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-06-27T21:50:21.444 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:50:21.445 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:50:21.445 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:50:21.445 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:50:21.445 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:50:21.445 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63994 2024-06-27T21:50:21.445 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63994/$cluster-$name.asok' 2024-06-27T21:50:21.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.63994/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:50:21.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:50:21.648 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:50:21.648 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:50:21.649 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:50:21.649 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:50:21.649 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:50:21.649 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:50:21.650 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:50:21.650 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:50:21.650 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:50:21.650 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:50:21.650 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63994 2024-06-27T21:50:21.650 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.63994/ceph-mon.a.asok 2024-06-27T21:50:21.650 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:50:21.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.63994/ceph-mon.a.asok config get fsid 2024-06-27T21:50:21.768 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:50:21.768 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:50:21.768 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:50:21.768 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:50:21.769 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:50:21.769 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:50:21.769 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:50:21.769 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:50:21.770 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:50:21.770 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:50:21.770 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63994 2024-06-27T21:50:21.770 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.63994/ceph-mon.a.asok 2024-06-27T21:50:21.771 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:50:21.771 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.63994/ceph-mon.a.asok config get mon_host 2024-06-27T21:50:21.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-06-27T21:50:21.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-06-27T21:50:21.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:50:21.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:50:21.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:50:21.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2024-06-27T21:50:21.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10fb0861-c531-441c-adaf-3e91d91d97be --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-06-27T21:50:21.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:50:21.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:50:21.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:50:21.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63994 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63994/$cluster-$name.asok' 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63994/$cluster-$name.asok' 2024-06-27T21:50:21.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:50:21.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:50:21.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:50:21.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-06-27T21:50:21.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-06-27T21:50:21.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:50:21.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:50:21.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:50:21.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:50:21.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:50:21.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:50:21.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2024-06-27T21:50:21.889 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:50:21.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f10b4784-697d-4612-94dc-76a3056f0ff1 2024-06-27T21:50:21.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f10b4784-697d-4612-94dc-76a3056f0ff1' 2024-06-27T21:50:21.889 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 f10b4784-697d-4612-94dc-76a3056f0ff1 2024-06-27T21:50:21.889 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:50:21.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCd3n1mDx2+NRAA7Yl4RfxK1a4K+ABPke01qg== 2024-06-27T21:50:21.904 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCd3n1mDx2+NRAA7Yl4RfxK1a4K+ABPke01qg=="}' 2024-06-27T21:50:21.904 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f10b4784-697d-4612-94dc-76a3056f0ff1 -i td/rados-striper/0/new.json 2024-06-27T21:50:22.908 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:50:22.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2024-06-27T21:50:22.929 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=10fb0861-c531-441c-adaf-3e91d91d97be --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63994/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCd3n1mDx2+NRAA7Yl4RfxK1a4K+ABPke01qg== --osd-uuid f10b4784-697d-4612-94dc-76a3056f0ff1 2024-06-27T21:50:22.963 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:22.956+0000 7f74225847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:50:22.985 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:22.980+0000 7f74225847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:50:22.987 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:22.980+0000 7f74225847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:50:22.988 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:22.980+0000 7f74225847c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-06-27T21:50:22.988 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:22.980+0000 7f74225847c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-06-27T21:50:22.989 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:22.980+0000 7f74225847c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-06-27T21:50:23.025 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:23.020+0000 7f74225847c0 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-06-27T21:50:29.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2024-06-27T21:50:29.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:50:29.209 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:50:29.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:50:29.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:50:29.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:50:29.466 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:50:29.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:50:29.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=10fb0861-c531-441c-adaf-3e91d91d97be --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63994/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:50:29.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:50:29.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:50:29.498 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:29.492+0000 7fba9ac037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:50:29.500 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:29.492+0000 7fba9ac037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:50:29.501 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:29.496+0000 7fba9ac037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:50:29.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:50:29.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:50:29.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:50:29.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:50:29.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:50:29.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:29.699 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:50:29.699 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:50:29.699 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:29.699 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:29.929 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:30.055 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:30.048+0000 7fba9ac037c0 -1 Falling back to public interface 2024-06-27T21:50:30.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:30.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:30.932 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:50:30.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:50:30.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:30.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:31.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:32.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:32.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:32.170 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:50:32.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:50:32.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:32.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:32.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:32.738 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:32.732+0000 7fba9ac037c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:50:33.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:33.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:33.409 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:50:33.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:50:33.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:33.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:33.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:34.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:34.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:34.848 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:50:34.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:50:34.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:34.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:35.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:36.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:36.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:36.087 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:50:36.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:50:36.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:36.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:36.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:37.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:37.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:37.326 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:50:37.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:50:37.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:37.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:37.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:38.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:38.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:38.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:50:38.565 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:50:38.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:38.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:38.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:39.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:39.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:39.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:50:39.806 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:50:39.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:39.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:40.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:41.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:41.051 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:50:41.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:41.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:50:41.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:41.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:41.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:42.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:42.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:42.292 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:50:42.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:50:42.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:42.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:42.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:43.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:43.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:43.539 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:50:43.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:50:43.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:43.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:43.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:44.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:44.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:44.779 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:50:44.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:50:44.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:44.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:45.018 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:46.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:46.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:46.020 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:50:46.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:50:46.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:46.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:46.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:47.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:47.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:47.261 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:50:47.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:50:47.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:47.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:47.496 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:48.497 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:48.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:48.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:50:48.498 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:50:48.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:48.499 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:48.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:49.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:49.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:49.738 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:50:49.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:50:49.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:49.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:49.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:50.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:50.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:50.990 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:50:50.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:50:50.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:50.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:51.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:52.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:52.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:52.219 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T21:50:52.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T21:50:52.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:52.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:52.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:53.462 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:53.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:53.463 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T21:50:53.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T21:50:53.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:53.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:53.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:54.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:54.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:54.702 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T21:50:54.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T21:50:54.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:54.706 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:54.941 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:50:55.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:50:55.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:50:55.943 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T21:50:55.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T21:50:55.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:50:55.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:50:56.178 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3461135883,v1:127.0.0.1:6801/3461135883] [v2:127.0.0.1:6802/3461135883,v1:127.0.0.1:6803/3461135883] exists,up f10b4784-697d-4612-94dc-76a3056f0ff1 2024-06-27T21:50:56.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:50:56.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:50:56.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:50:56.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-06-27T21:50:56.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-06-27T21:50:56.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:50:56.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T21:50:56.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:50:56.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2024-06-27T21:50:56.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10fb0861-c531-441c-adaf-3e91d91d97be --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:50:56.182 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:50:56.183 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63994 2024-06-27T21:50:56.183 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63994/$cluster-$name.asok' 2024-06-27T21:50:56.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63994/$cluster-$name.asok' 2024-06-27T21:50:56.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:50:56.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:50:56.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:50:56.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-06-27T21:50:56.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-06-27T21:50:56.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:50:56.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:50:56.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:50:56.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:50:56.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:50:56.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:50:56.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2024-06-27T21:50:56.184 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:50:56.184 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 6e7999e7-fde6-4445-b28a-243f9cb5a673 2024-06-27T21:50:56.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6e7999e7-fde6-4445-b28a-243f9cb5a673 2024-06-27T21:50:56.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6e7999e7-fde6-4445-b28a-243f9cb5a673' 2024-06-27T21:50:56.185 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:50:56.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDA3n1moHY2DBAAhUjVlwxk/8xLckYXF5/eQw== 2024-06-27T21:50:56.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDA3n1moHY2DBAAhUjVlwxk/8xLckYXF5/eQw=="}' 2024-06-27T21:50:56.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6e7999e7-fde6-4445-b28a-243f9cb5a673 -i td/rados-striper/1/new.json 2024-06-27T21:50:56.571 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:50:56.589 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2024-06-27T21:50:56.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=10fb0861-c531-441c-adaf-3e91d91d97be --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63994/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDA3n1moHY2DBAAhUjVlwxk/8xLckYXF5/eQw== --osd-uuid 6e7999e7-fde6-4445-b28a-243f9cb5a673 2024-06-27T21:50:56.624 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:56.620+0000 7f22171fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:50:56.698 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:56.696+0000 7f22171fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:50:56.700 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:56.696+0000 7f22171fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:50:56.701 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:56.700+0000 7f22171fd7c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-06-27T21:50:56.701 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:56.700+0000 7f22171fd7c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-06-27T21:50:56.701 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:56.700+0000 7f22171fd7c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-06-27T21:50:56.823 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:50:56.820+0000 7f22171fd7c0 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-06-27T21:51:03.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2024-06-27T21:51:03.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:51:03.499 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T21:51:03.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T21:51:03.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:51:03.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T21:51:03.749 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T21:51:03.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=10fb0861-c531-441c-adaf-3e91d91d97be --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63994/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:51:03.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:51:03.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:51:03.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:51:03.780 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:03.780+0000 7ff20aa407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:51:03.783 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:03.780+0000 7ff20aa407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:51:03.785 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:03.784+0000 7ff20aa407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:51:03.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T21:51:03.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:51:03.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T21:51:03.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:51:03.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:51:03.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:03.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:51:03.980 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:51:03.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:03.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:04.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:04.327 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:04.324+0000 7ff20aa407c0 -1 Falling back to public interface 2024-06-27T21:51:05.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:05.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:05.209 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:51:05.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:51:05.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:05.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:05.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:06.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:06.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:06.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:51:06.449 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:51:06.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:06.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:06.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:07.025 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:07.024+0000 7ff20aa407c0 -1 osd.1 0 log_to_monitors true 2024-06-27T21:51:07.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:07.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:07.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:51:07.679 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:51:07.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:07.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:08.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:09.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:09.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:09.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:51:09.170 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:51:09.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:09.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:09.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:10.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:10.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:10.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:51:10.417 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:51:10.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:10.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:10.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:51:11.658 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:51:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:11.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:12.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:12.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:12.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:51:12.899 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:51:12.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:12.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:13.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:14.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:14.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:14.138 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:51:14.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:51:14.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:14.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:14.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:15.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:15.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:15.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:51:15.380 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:51:15.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:15.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:15.619 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:16.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:16.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:16.621 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:51:16.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:51:16.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:16.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:16.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:17.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:17.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:17.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:51:17.858 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:51:17.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:17.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:18.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:19.097 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:19.097 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:19.098 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:51:19.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:51:19.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:19.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:19.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:20.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:20.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:20.336 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:51:20.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:51:20.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:20.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:20.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:21.580 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:21.580 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:21.580 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:51:21.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:51:21.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:21.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:21.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:22.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:22.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:22.816 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:51:22.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:51:22.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:22.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:23.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:24.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:24.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:24.058 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:51:24.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:51:24.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:24.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:24.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:25.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:25.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:25.294 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:51:25.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:51:25.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:25.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:25.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:26.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:26.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:26.540 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T21:51:26.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T21:51:26.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:26.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:26.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:27.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:27.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:27.779 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T21:51:27.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T21:51:27.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:27.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:28.014 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:29.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:29.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:29.016 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T21:51:29.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T21:51:29.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:29.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:51:29.330 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/1804425439,v1:127.0.0.1:6809/1804425439] [v2:127.0.0.1:6810/1804425439,v1:127.0.0.1:6811/1804425439] exists,up 6e7999e7-fde6-4445-b28a-243f9cb5a673 2024-06-27T21:51:29.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10fb0861-c531-441c-adaf-3e91d91d97be --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:51:29.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:51:29.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-06-27T21:51:29.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-06-27T21:51:29.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:51:29.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:51:29.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-06-27T21:51:29.334 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:51:29.334 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:51:29.334 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:51:29.336 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:51:29.336 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:51:29.336 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63994 2024-06-27T21:51:29.336 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63994/$cluster-$name.asok' 2024-06-27T21:51:29.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63994/$cluster-$name.asok' 2024-06-27T21:51:29.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2024-06-27T21:51:29.337 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:51:29.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=474d691e-3392-4bc7-8ddc-be0ff9bd927f 2024-06-27T21:51:29.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 474d691e-3392-4bc7-8ddc-be0ff9bd927f' 2024-06-27T21:51:29.338 INFO:tasks.workunit.client.0.smithi042.stdout:add osd2 474d691e-3392-4bc7-8ddc-be0ff9bd927f 2024-06-27T21:51:29.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:51:29.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDh3n1mypsSFRAAE8B/l/AlnNQM9k8PE9rhYQ== 2024-06-27T21:51:29.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDh3n1mypsSFRAAE8B/l/AlnNQM9k8PE9rhYQ=="}' 2024-06-27T21:51:29.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 474d691e-3392-4bc7-8ddc-be0ff9bd927f -i td/rados-striper/2/new.json 2024-06-27T21:51:29.715 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:51:29.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2024-06-27T21:51:29.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=10fb0861-c531-441c-adaf-3e91d91d97be --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63994/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDh3n1mypsSFRAAE8B/l/AlnNQM9k8PE9rhYQ== --osd-uuid 474d691e-3392-4bc7-8ddc-be0ff9bd927f 2024-06-27T21:51:29.763 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:29.760+0000 7fe62efe77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:51:29.766 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:29.764+0000 7fe62efe77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:51:29.767 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:29.764+0000 7fe62efe77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:51:29.768 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:29.764+0000 7fe62efe77c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-06-27T21:51:29.768 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:29.764+0000 7fe62efe77c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-06-27T21:51:29.768 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:29.764+0000 7fe62efe77c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-06-27T21:51:29.841 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:29.840+0000 7fe62efe77c0 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-06-27T21:51:36.018 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2024-06-27T21:51:36.018 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:51:36.020 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd2 key to auth repository 2024-06-27T21:51:36.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-27T21:51:36.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:51:36.273 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T21:51:36.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-27T21:51:36.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:51:36.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:51:36.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=10fb0861-c531-441c-adaf-3e91d91d97be --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63994/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:51:36.275 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:51:36.306 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:36.304+0000 7f007474f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:51:36.308 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:36.308+0000 7f007474f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:51:36.310 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:36.308+0000 7f007474f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:51:36.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-27T21:51:36.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:51:36.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T21:51:36.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:51:36.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:51:36.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:36.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:51:36.507 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:51:36.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:36.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:36.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:36.871 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:36.868+0000 7f007474f7c0 -1 Falling back to public interface 2024-06-27T21:51:37.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:37.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:37.740 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:51:37.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:51:37.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:37.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:37.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:38.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:38.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:38.977 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:51:38.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:51:38.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:38.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:39.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:39.560 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:51:39.556+0000 7f007474f7c0 -1 osd.2 0 log_to_monitors true 2024-06-27T21:51:40.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:40.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:40.214 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:51:40.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:51:40.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:40.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:40.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:41.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:41.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:41.513 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:51:41.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:51:41.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:41.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:41.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:42.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:42.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:42.756 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:51:42.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:51:42.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:42.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:42.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:44.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:44.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:44.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:51:44.001 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:51:44.002 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:44.002 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:44.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:45.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:45.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:45.242 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:51:45.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:51:45.243 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:45.243 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:45.481 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:46.482 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:46.482 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:46.482 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:51:46.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:51:46.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:46.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:46.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:47.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:47.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:47.728 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:51:47.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:51:47.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:47.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:47.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:48.968 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:48.968 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:48.968 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:51:48.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:51:48.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:48.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:49.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:50.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:50.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:50.211 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:51:50.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:51:50.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:50.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:50.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:51.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:51.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:51.443 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:51:51.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:51:51.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:51.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:51.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:52.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:52.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:52.681 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:51:52.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:51:52.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:52.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:52.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:53.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:53.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:53.924 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:51:53.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:51:53.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:53.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:54.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:55.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:55.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:55.166 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:51:55.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:51:55.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:55.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:55.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:56.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:56.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:56.406 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:51:56.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:51:56.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:56.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:56.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:57.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:57.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:57.647 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:51:57.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:51:57.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:57.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:57.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:51:58.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:51:58.890 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:51:58.890 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T21:51:58.890 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T21:51:58.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:51:58.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:51:59.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:00.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:00.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:00.131 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T21:52:00.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T21:52:00.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:00.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:52:00.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:01.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:01.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:01.374 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T21:52:01.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T21:52:01.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:01.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:52:01.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:02.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:02.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:02.615 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T21:52:02.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T21:52:02.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:02.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T21:52:02.884 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/1853113309,v1:127.0.0.1:6817/1853113309] [v2:127.0.0.1:6818/1853113309,v1:127.0.0.1:6819/1853113309] exists,up 474d691e-3392-4bc7-8ddc-be0ff9bd927f 2024-06-27T21:52:02.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:52:02.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:52:02.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:52:02.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-06-27T21:52:02.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T21:52:03.106 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T21:52:03.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T21:52:03.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T21:52:03.470 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T21:52:03.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T21:52:04.489 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T21:52:06.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-06-27T21:52:06.840 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records in 2024-06-27T21:52:06.840 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records out 2024-06-27T21:52:06.840 INFO:tasks.workunit.client.0.smithi042.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 0.000209299 s, 5.9 MB/s 2024-06-27T21:52:06.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-06-27T21:52:07.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-06-27T21:52:07.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-06-27T21:52:07.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-06-27T21:52:07.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-06-27T21:52:07.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-06-27T21:52:07.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-06-27T21:52:07.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-06-27T21:52:07.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-06-27T21:52:07.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-06-27T21:52:07.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-06-27T21:52:07.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-06-27T21:52:07.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-06-27T21:52:07.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-06-27T21:52:07.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-06-27T21:52:08.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-06-27T21:52:08.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-06-27T21:52:08.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-06-27T21:52:08.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-06-27T21:52:08.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-06-27T21:52:08.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-06-27T21:52:08.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-06-27T21:52:08.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-06-27T21:52:08.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-06-27T21:52:08.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-06-27T21:52:08.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-06-27T21:52:08.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-06-27T21:52:08.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-06-27T21:52:08.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-06-27T21:52:08.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-06-27T21:52:08.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-06-27T21:52:08.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-06-27T21:52:08.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-06-27T21:52:08.714 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-06-27T21:52:08.714 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-06-27T21:52:08.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-06-27T21:52:08.846 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-06-27T21:52:08.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-06-27T21:52:08.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-06-27T21:52:08.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/rados-striper 2024-06-27T21:52:08.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-06-27T21:52:08.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local 'expected=No data available' 2024-06-27T21:52:08.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-06-27T21:52:08.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-06-27T21:52:08.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-06-27T21:52:08.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-06-27T21:52:08.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-06-27T21:52:08.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-06-27T21:52:08.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-06-27T21:52:08.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-06-27T21:52:08.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/rados-striper 2024-06-27T21:52:08.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-06-27T21:52:08.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local 'expected=No data available' 2024-06-27T21:52:08.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-06-27T21:52:08.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-06-27T21:52:08.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-06-27T21:52:08.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-06-27T21:52:08.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-06-27T21:52:08.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-06-27T21:52:08.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-06-27T21:52:08.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-06-27T21:52:09.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-06-27T21:52:09.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/rados-striper 2024-06-27T21:52:09.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-06-27T21:52:09.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local 'expected=No such file or directory' 2024-06-27T21:52:09.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-06-27T21:52:09.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-06-27T21:52:09.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: rados --pool rbd --striper stat toyfile 2024-06-27T21:52:09.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-06-27T21:52:09.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-06-27T21:52:09.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-06-27T21:52:09.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-06-27T21:52:09.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-06-27T21:52:09.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/rados-striper 2024-06-27T21:52:09.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-06-27T21:52:09.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local 'expected=No such file or directory' 2024-06-27T21:52:09.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-06-27T21:52:09.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-06-27T21:52:09.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-06-27T21:52:09.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-06-27T21:52:09.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-06-27T21:52:09.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-06-27T21:52:09.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-06-27T21:52:09.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-06-27T21:52:09.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-06-27T21:52:09.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:52:09.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-06-27T21:52:09.193 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:52:09.193 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:52:09.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:52:09.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:52:09.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:52:09.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:52:09.501 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:52:09.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:52:09.504 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:52:09.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:52:09.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:52:09.506 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:52:09.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:52:09.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:52:09.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:52:09.508 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:52:09.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:52:09.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:52:09.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-06-27T21:52:09.528 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:52:09.528 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:09.528 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63994 2024-06-27T21:52:09.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.63994 2024-06-27T21:52:09.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:52:09.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:52:09.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-06-27T21:52:09.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/rados-striper 0 2024-06-27T21:52:09.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-06-27T21:52:09.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-27T21:52:09.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-06-27T21:52:09.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:52:09.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:52:09.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:52:09.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:52:09.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:52:09.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:52:09.534 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:52:09.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:52:09.535 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:52:09.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:52:09.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:52:09.537 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:52:09.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:52:09.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:52:09.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:52:09.539 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:52:09.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:52:09.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-27T21:52:09.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-06-27T21:52:09.543 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:52:09.543 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:09.543 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63994 2024-06-27T21:52:09.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.63994 2024-06-27T21:52:09.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:52:09.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:52:09.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-06-27T21:52:09.546 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-27T21:52:09.546 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-27T21:52:09.601 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-06-27T21:52:09.601 DEBUG:teuthology.orchestra.run.smithi042:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-06-27T21:52:09.659 INFO:tasks.workunit.client.0.smithi042.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-27T21:52:09.664 INFO:tasks.workunit.client.0.smithi042.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-27T21:52:09.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: run_tests: export .:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T21:52:09.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2344: export: `.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin': not a valid identifier 2024-06-27T21:52:09.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: run_tests: export CEPH_MON=127.0.0.1:7109 2024-06-27T21:52:09.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: run_tests: CEPH_MON=127.0.0.1:7109 2024-06-27T21:52:09.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: export CEPH_ARGS 2024-06-27T21:52:09.664 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: uuidgen 2024-06-27T21:52:09.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: CEPH_ARGS+=' --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none ' 2024-06-27T21:52:09.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-06-27T21:52:09.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: run_tests: export CEPH_CONF=/dev/null 2024-06-27T21:52:09.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: run_tests: CEPH_CONF=/dev/null 2024-06-27T21:52:09.666 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: run_tests: set 2024-06-27T21:52:09.666 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-06-27T21:52:09.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: run_tests: local 'funcs=test_activate_osd 2024-06-27T21:52:09.670 INFO:tasks.workunit.client.0.smithi042.stderr:test_activate_osd_after_mark_down 2024-06-27T21:52:09.670 INFO:tasks.workunit.client.0.smithi042.stderr:test_activate_osd_skip_benchmark 2024-06-27T21:52:09.670 INFO:tasks.workunit.client.0.smithi042.stderr:test_destroy_osd 2024-06-27T21:52:09.670 INFO:tasks.workunit.client.0.smithi042.stderr:test_display_logs 2024-06-27T21:52:09.670 INFO:tasks.workunit.client.0.smithi042.stderr:test_erasure_code_plugin_exists 2024-06-27T21:52:09.670 INFO:tasks.workunit.client.0.smithi042.stderr:test_expect_failure 2024-06-27T21:52:09.670 INFO:tasks.workunit.client.0.smithi042.stderr:test_flush_pg_stats 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_config 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_is_making_recovery_progress 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_last_scrub_stamp 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_not_primary 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_num_active_clean 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_num_active_or_peered 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_num_pgs 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_op_scheduler 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_osd_id_used_by_pgs 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_osds 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_pg 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_primary 2024-06-27T21:52:09.671 INFO:tasks.workunit.client.0.smithi042.stderr:test_get_timeout_delays 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_is_clean 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_kill_daemon 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_kill_daemons 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_objectstore_tool 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_pg_schedule_scrub 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_pg_scrub 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_repair 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_run_in_background 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_run_mon 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_run_osd 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_set_config 2024-06-27T21:52:09.672 INFO:tasks.workunit.client.0.smithi042.stderr:test_setup 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:test_teardown 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:test_wait_background 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:test_wait_for_clean 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:test_wait_for_health_ok 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:test_wait_for_osd 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:test_wait_for_peered 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:test_wait_for_scrub 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:test_wait_osd_id_used_by_pgs' 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: run_tests: local dir=td/ceph-helpers 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_activate_osd td/ceph-helpers 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2024-06-27T21:52:09.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:52:09.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:52:09.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:52:09.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:52:09.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:52:09.674 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:52:09.674 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:52:09.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:52:09.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:52:09.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:52:09.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:52:09.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:52:09.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:52:09.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:52:09.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:52:09.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:52:09.676 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:52:09.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:52:09.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:52:09.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:52:09.678 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:52:09.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:52:09.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:52:09.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:52:09.681 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:52:09.681 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:09.681 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:09.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:52:09.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:52:09.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:52:09.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:52:09.684 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:52:09.684 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:09.684 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:09.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:52:09.686 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:52:09.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-27T21:52:09.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-27T21:52:09.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:52:09.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:52:09.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2024-06-27T21:52:09.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:52:09.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:52:09.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:52:09.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:52:09.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:52:09.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-27T21:52:10.186 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:52:10.186 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:52:10.186 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:52:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:52:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:10.187 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:52:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:52:10.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:52:10.397 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:52:10.397 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:52:10.397 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:52:10.397 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:52:10.398 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:52:10.398 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:52:10.398 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:52:10.398 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:52:10.398 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:52:10.398 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:10.398 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:10.399 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:52:10.399 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:52:10.399 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:52:10.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:52:10.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:52:10.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:52:10.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:52:10.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:52:10.518 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:52:10.519 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:52:10.519 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:52:10.519 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:52:10.519 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:10.519 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:10.519 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:52:10.519 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:52:10.519 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:52:10.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2024-06-27T21:52:10.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:52:10.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:52:10.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:52:10.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:52:10.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:52:10.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:52:11.586 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:52:11.586 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:52:11.586 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:52:11.587 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:52:11.587 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:11.587 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:11.587 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:52:11.588 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:52:11.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:52:11.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2024-06-27T21:52:11.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T21:52:11.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:52:11.657 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:52:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:52:11.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:52:11.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:52:11.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:52:11.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:52:11.659 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:52:11.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=069a869f-a4ab-4d0f-ab22-bd8167dfad4d 2024-06-27T21:52:11.659 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 069a869f-a4ab-4d0f-ab22-bd8167dfad4d 2024-06-27T21:52:11.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 069a869f-a4ab-4d0f-ab22-bd8167dfad4d' 2024-06-27T21:52:11.660 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:52:11.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAL331mP353KBAAPlxI3qgSl8MnMdsw/TOnJQ== 2024-06-27T21:52:11.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAL331mP353KBAAPlxI3qgSl8MnMdsw/TOnJQ=="}' 2024-06-27T21:52:11.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 069a869f-a4ab-4d0f-ab22-bd8167dfad4d -i td/ceph-helpers/0/new.json 2024-06-27T21:52:12.018 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:52:12.034 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T21:52:12.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAL331mP353KBAAPlxI3qgSl8MnMdsw/TOnJQ== --osd-uuid 069a869f-a4ab-4d0f-ab22-bd8167dfad4d 2024-06-27T21:52:12.078 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:12.075+0000 7fda685567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:12.094 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:12.091+0000 7fda685567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:12.096 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:12.091+0000 7fda685567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:12.096 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:12.095+0000 7fda685567c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:52:12.096 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:12.095+0000 7fda685567c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:52:12.096 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:12.095+0000 7fda685567c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:52:12.169 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:12.167+0000 7fda685567c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T21:52:18.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T21:52:18.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:52:18.755 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:52:18.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:52:18.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:52:19.349 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:52:19.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:52:19.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:52:19.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:52:19.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:52:19.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:52:19.382 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:19.379+0000 7f50abdb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:19.401 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:19.399+0000 7f50abdb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:19.402 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:19.399+0000 7f50abdb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:19.785 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:52:19.785 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:52:19.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:52:19.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:52:19.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:52:19.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:19.786 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:52:19.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:52:19.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:19.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:19.966 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:19.963+0000 7f50abdb27c0 -1 Falling back to public interface 2024-06-27T21:52:20.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:21.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:21.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:21.229 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:52:21.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:52:21.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:21.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:21.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:22.592 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:22.587+0000 7f50abdb27c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:52:22.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:22.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:22.709 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:52:22.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:52:22.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:22.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:23.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:24.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:24.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:24.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:52:24.303 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:52:24.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:24.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:24.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:25.774 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:25.774 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:25.774 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:52:25.774 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:52:25.775 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:25.775 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:26.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:27.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:27.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:27.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:52:27.267 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:52:27.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:27.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:27.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:28.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:28.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:28.831 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:52:28.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:52:28.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:28.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:29.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:30.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:30.445 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:52:30.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:30.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:52:30.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:30.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:30.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:31.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:31.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:31.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:52:31.894 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:52:31.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:31.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:32.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:33.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:33.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:33.445 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:52:33.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:52:33.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:33.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:33.908 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:34.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:34.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:34.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:52:34.910 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:52:34.910 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:34.910 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:35.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:36.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:36.521 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:52:36.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:36.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:52:36.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:36.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:37.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:38.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:38.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:38.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:52:38.324 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:52:38.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:38.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:39.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:40.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:40.660 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:52:40.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:40.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:52:40.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:40.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:41.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:42.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:42.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:42.222 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:52:42.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:52:42.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:42.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:43.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:44.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:44.022 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:52:44.023 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:44.023 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:52:44.023 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:44.023 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:45.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:46.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:46.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:46.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:52:46.065 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:52:46.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:46.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:46.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:47.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:47.988 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:52:47.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:47.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:52:47.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:47.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:48.421 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3047608415,v1:127.0.0.1:6803/3047608415] [v2:127.0.0.1:6804/3047608415,v1:127.0.0.1:6805/3047608415] exists,up 069a869f-a4ab-4d0f-ab22-bd8167dfad4d 2024-06-27T21:52:48.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:52:48.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:52:48.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:52:48.422 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2024-06-27T21:52:48.423 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:52:48.423 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:52:48.423 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:52:48.423 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:48.423 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:48.423 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.0.asok 2024-06-27T21:52:48.423 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2024-06-27T21:52:48.423 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-osd.0.asok config get osd_max_backfills 2024-06-27T21:52:48.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-06-27T21:52:48.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-06-27T21:52:48.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2024-06-27T21:52:48.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-06-27T21:52:48.550 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:52:48.550 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:52:48.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:52:48.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:52:48.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:52:48.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:52:48.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-06-27T21:52:48.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-27T21:52:48.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-27T21:52:48.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:52:48.862 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-27T21:52:48.863 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:52:48.863 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:52:48.864 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:52:48.864 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:48.864 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:48.864 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:52:48.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:52:48.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:52:48.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:52:48.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:52:48.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:52:48.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:52:48.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:52:48.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:52:48.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-27T21:52:48.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-06-27T21:52:48.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:52:48.865 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:52:48.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-27T21:52:48.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-06-27T21:52:48.867 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-06-27T21:52:48.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-27T21:52:48.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-27T21:52:48.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-27T21:52:48.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-27T21:52:48.901 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:48.899+0000 7f79a21367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:48.904 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:48.899+0000 7f79a21367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:48.905 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:48.903+0000 7f79a21367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:49.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-27T21:52:49.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:52:49.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:52:49.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:52:49.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:52:49.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:49.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:52:49.365 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:52:49.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:49.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:49.465 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:49.463+0000 7f79a21367c0 -1 Falling back to public interface 2024-06-27T21:52:49.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:50.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:50.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:50.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:52:50.850 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:52:50.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:50.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:51.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:52.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:52.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:52.324 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:52:52.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:52:52.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:52.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:52.466 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:52.463+0000 7f79a21367c0 -1 osd.0 6 log_to_monitors true 2024-06-27T21:52:52.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:52:53.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:52:53.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:52:53.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:52:53.791 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:52:53.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:52:53.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:52:54.272 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/532191685,v1:127.0.0.1:6803/532191685] [v2:127.0.0.1:6804/532191685,v1:127.0.0.1:6805/532191685] exists,up 069a869f-a4ab-4d0f-ab22-bd8167dfad4d 2024-06-27T21:52:54.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:52:54.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:52:54.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:52:54.273 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2024-06-27T21:52:54.273 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:52:54.273 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:52:54.274 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:52:54.274 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:54.274 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:54.274 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.0.asok 2024-06-27T21:52:54.275 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2024-06-27T21:52:54.275 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-osd.0.asok config get osd_max_backfills 2024-06-27T21:52:54.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-06-27T21:52:54.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-06-27T21:52:54.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2024-06-27T21:52:54.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:52:54.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:52:54.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:52:54.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:52:54.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:52:54.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:52:54.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:52:54.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:52:54.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:52:54.702 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:52:54.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:52:54.704 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:52:54.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:52:54.706 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:52:54.706 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:52:54.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:52:54.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:52:54.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:52:54.708 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:52:54.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:52:54.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:52:54.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:52:54.719 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:52:54.719 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:54.720 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:54.720 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:52:54.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:52:54.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:52:54.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:52:54.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:52:54.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:52:54.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:52:54.725 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:52:54.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:52:54.726 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:52:54.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:52:54.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:52:54.728 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:52:54.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:52:54.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:52:54.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:52:54.729 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:52:54.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:52:54.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:52:54.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:52:54.733 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:52:54.733 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:54.733 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:54.733 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:52:54.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:52:54.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:52:54.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:52:54.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:52:54.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:54.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:54.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:52:54.738 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:52:54.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:52:54.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:52:54.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:52:54.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-06-27T21:52:54.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:52:54.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:52:54.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:52:54.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:52:54.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:52:54.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-27T21:52:55.264 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:52:55.264 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:52:55.264 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:52:55.265 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:52:55.265 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:55.265 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:55.265 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:52:55.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:52:55.477 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:52:55.478 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:52:55.478 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:52:55.478 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:52:55.478 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:52:55.479 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:52:55.479 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:52:55.479 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:52:55.479 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:52:55.480 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:52:55.480 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:55.480 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:55.480 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:52:55.480 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:52:55.480 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:52:55.601 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:52:55.601 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:52:55.601 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:52:55.601 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:52:55.602 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:52:55.602 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:52:55.603 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:52:55.603 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:52:55.603 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:52:55.603 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:55.603 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:55.603 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:52:55.603 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:52:55.603 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:52:55.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-06-27T21:52:55.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:52:55.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:52:55.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:52:55.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:52:55.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:52:55.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:52:56.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:52:56.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:52:56.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:52:56.641 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:52:56.641 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:56.641 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:56.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:52:56.642 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:52:56.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:52:56.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-06-27T21:52:56.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:52:56.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:52:56.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:52:56.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:52:56.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:52:56.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:52:56.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:52:56.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:52:56.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:52:56.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:52:56.685 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:52:56.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=317468aa-c603-4786-ae73-7bc9e0ff667d 2024-06-27T21:52:56.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 317468aa-c603-4786-ae73-7bc9e0ff667d' 2024-06-27T21:52:56.685 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 317468aa-c603-4786-ae73-7bc9e0ff667d 2024-06-27T21:52:56.686 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:52:56.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4331mZUy7KRAAks+7Tkg1bfTzI0MtYdPTJg== 2024-06-27T21:52:56.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4331mZUy7KRAAks+7Tkg1bfTzI0MtYdPTJg=="}' 2024-06-27T21:52:56.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 317468aa-c603-4786-ae73-7bc9e0ff667d -i td/ceph-helpers/0/new.json 2024-06-27T21:52:57.037 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:52:57.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T21:52:57.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4331mZUy7KRAAks+7Tkg1bfTzI0MtYdPTJg== --osd-uuid 317468aa-c603-4786-ae73-7bc9e0ff667d 2024-06-27T21:52:57.091 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:57.087+0000 7f5a428a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:57.113 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:57.111+0000 7f5a428a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:57.115 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:57.111+0000 7f5a428a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:52:57.116 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:57.111+0000 7f5a428a07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:52:57.116 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:57.111+0000 7f5a428a07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:52:57.116 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:57.111+0000 7f5a428a07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:52:57.154 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:52:57.151+0000 7f5a428a07c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T21:53:03.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T21:53:03.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:53:03.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:53:03.756 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:53:03.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:53:04.378 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:53:04.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:53:04.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:53:04.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:53:04.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:53:04.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:53:04.411 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:04.407+0000 7fb6c166a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:04.445 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:04.443+0000 7fb6c166a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:04.447 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:04.443+0000 7fb6c166a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:04.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:53:04.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:53:04.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:53:04.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:53:04.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:53:04.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:04.834 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:53:04.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:53:04.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:04.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:05.022 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:05.019+0000 7fb6c166a7c0 -1 Falling back to public interface 2024-06-27T21:53:05.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:06.285 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:06.285 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:06.285 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:53:06.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:53:06.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:06.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:06.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:07.705 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:07.703+0000 7fb6c166a7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:53:07.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:07.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:07.781 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:53:07.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:53:07.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:07.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:08.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:09.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:09.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:09.431 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:53:09.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:53:09.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:09.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:09.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:10.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:10.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:10.895 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:53:10.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:53:10.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:10.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:11.330 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:12.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:12.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:12.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:53:12.333 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:53:12.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:12.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:12.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:13.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:13.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:13.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:53:13.899 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:53:13.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:13.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:14.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:15.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:15.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:15.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:53:15.364 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:53:15.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:15.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:15.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:16.927 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:16.927 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:16.927 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:53:16.927 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:53:16.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:16.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:17.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:18.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:18.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:18.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:53:18.442 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:53:18.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:18.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:18.929 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:19.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:19.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:19.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:53:19.932 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:53:19.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:19.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:20.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:21.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:21.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:21.388 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:53:21.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:53:21.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:21.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:22.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:23.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:23.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:23.394 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:53:23.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:53:23.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:23.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:23.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:24.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:24.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:24.999 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:53:24.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:53:24.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:24.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:26.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:27.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:27.096 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:53:27.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:27.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:53:27.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:27.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:28.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:29.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:29.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:29.532 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:53:29.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:53:29.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:29.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:30.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:31.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:31.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:31.575 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:53:31.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:53:31.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:31.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:32.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:33.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:33.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:33.028 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:53:33.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:53:33.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:33.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:33.465 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1302580025,v1:127.0.0.1:6803/1302580025] [v2:127.0.0.1:6804/1302580025,v1:127.0.0.1:6805/1302580025] exists,up 317468aa-c603-4786-ae73-7bc9e0ff667d 2024-06-27T21:53:33.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:53:33.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:53:33.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:53:33.467 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-06-27T21:53:33.467 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:53:33.467 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:53:33.467 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:53:33.467 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:33.467 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:33.468 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.0.asok 2024-06-27T21:53:33.468 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-06-27T21:53:33.468 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-osd.0.asok config get osd_max_backfills 2024-06-27T21:53:33.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-06-27T21:53:33.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-06-27T21:53:33.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-06-27T21:53:33.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-06-27T21:53:33.594 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:53:33.594 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:53:33.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:53:33.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:53:33.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:53:33.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:53:33.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2024-06-27T21:53:34.373 INFO:tasks.workunit.client.0.smithi042.stderr:osd.0 is already down. 2024-06-27T21:53:34.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-06-27T21:53:34.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-06-27T21:53:34.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:53:34.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:53:34.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:53:34.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:34.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:53:34.391 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:53:34.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:34.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-06-27T21:53:34.818 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/1302580025,v1:127.0.0.1:6803/1302580025] [v2:127.0.0.1:6804/1302580025,v1:127.0.0.1:6805/1302580025] exists 317468aa-c603-4786-ae73-7bc9e0ff667d 2024-06-27T21:53:34.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:53:34.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:53:34.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:53:34.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-06-27T21:53:34.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-27T21:53:34.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-27T21:53:34.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:53:34.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:53:34.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:53:34.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:53:34.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:53:34.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-27T21:53:34.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-06-27T21:53:34.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:53:34.824 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:53:34.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-27T21:53:34.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-06-27T21:53:34.825 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-06-27T21:53:34.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-27T21:53:34.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-27T21:53:34.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-27T21:53:34.826 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-27T21:53:34.855 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:34.851+0000 7fd792f797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:34.858 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:34.855+0000 7fd792f797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:34.859 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:34.855+0000 7fd792f797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:35.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-27T21:53:35.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:53:35.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:53:35.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:53:35.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:53:35.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:35.248 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:53:35.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:53:35.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:35.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:35.406 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:35.403+0000 7fd792f797c0 -1 Falling back to public interface 2024-06-27T21:53:35.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:36.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:36.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:36.736 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:53:36.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:53:36.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:36.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:37.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:38.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:38.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:38.167 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:53:38.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:53:38.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:38.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:38.546 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:38.543+0000 7fd792f797c0 -1 osd.0 6 log_to_monitors true 2024-06-27T21:53:38.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:39.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:39.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:39.613 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:53:39.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:53:39.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:39.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:40.095 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/2330246876,v1:127.0.0.1:6803/2330246876] [v2:127.0.0.1:6804/2330246876,v1:127.0.0.1:6805/2330246876] exists,up 317468aa-c603-4786-ae73-7bc9e0ff667d 2024-06-27T21:53:40.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:53:40.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:53:40.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:53:40.097 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-06-27T21:53:40.097 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:53:40.097 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:53:40.098 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:53:40.098 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:40.098 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:40.098 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.0.asok 2024-06-27T21:53:40.098 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-06-27T21:53:40.098 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-osd.0.asok config get osd_max_backfills 2024-06-27T21:53:40.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-06-27T21:53:40.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-06-27T21:53:40.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-06-27T21:53:40.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:53:40.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:53:40.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:53:40.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:53:40.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:53:40.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:53:40.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:53:40.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:53:40.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:53:40.329 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:53:40.330 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:53:40.330 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:53:40.331 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:53:40.331 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:53:40.331 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:53:40.331 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:53:40.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:53:40.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:53:40.332 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:53:40.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:53:40.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:53:40.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:53:40.339 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:53:40.340 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:40.340 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:40.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:53:40.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:53:40.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:53:40.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:53:40.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-06-27T21:53:40.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-06-27T21:53:40.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-06-27T21:53:40.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:53:40.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:53:40.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:53:40.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:53:40.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:53:40.342 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:53:40.342 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:53:40.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:53:40.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:53:40.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:53:40.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:53:40.343 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:53:40.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:53:40.344 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:53:40.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:53:40.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:53:40.345 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:53:40.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:53:40.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:53:40.346 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:53:40.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:53:40.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:53:40.347 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:53:40.347 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:53:40.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:53:40.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:40.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:40.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:53:40.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:53:40.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:53:40.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:53:40.350 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:53:40.350 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:40.350 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:40.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:53:40.352 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:53:40.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:53:40.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:53:40.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:53:40.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-06-27T21:53:40.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:53:40.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:53:40.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:53:40.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:53:40.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:53:40.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-27T21:53:40.862 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:53:40.862 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:53:40.862 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:53:40.863 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:53:40.863 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:40.863 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:40.863 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:53:40.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:53:41.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:53:41.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:53:41.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:53:41.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:53:41.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:53:41.067 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:53:41.067 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:53:41.067 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:53:41.067 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:53:41.068 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:53:41.068 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:41.068 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:41.068 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:53:41.068 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:53:41.068 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:53:41.187 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:53:41.188 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:53:41.188 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:53:41.188 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:53:41.189 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:53:41.189 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:53:41.189 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:53:41.189 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:53:41.189 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:53:41.189 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:41.189 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:41.189 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:53:41.189 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:53:41.190 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:53:41.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-06-27T21:53:41.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:53:41.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:53:41.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:53:41.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:53:41.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:53:41.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:53:42.205 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:53:42.205 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:53:42.205 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:53:42.206 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:53:42.206 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:42.206 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:42.206 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:53:42.207 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:53:42.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:53:42.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-06-27T21:53:42.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T21:53:42.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:53:42.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:53:42.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:53:42.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:53:42.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:53:42.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:42.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-06-27T21:53:42.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:53:42.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:53:42.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63d5774e-76b2-49c0-b726-79d5d68d6341 2024-06-27T21:53:42.249 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 63d5774e-76b2-49c0-b726-79d5d68d6341 2024-06-27T21:53:42.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 63d5774e-76b2-49c0-b726-79d5d68d6341' 2024-06-27T21:53:42.250 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:53:42.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBm331meqvODxAAs651pUGt7bBRsVp3x24O1A== 2024-06-27T21:53:42.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBm331meqvODxAAs651pUGt7bBRsVp3x24O1A=="}' 2024-06-27T21:53:42.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63d5774e-76b2-49c0-b726-79d5d68d6341 -i td/ceph-helpers/0/new.json 2024-06-27T21:53:42.560 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:53:42.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T21:53:42.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQBm331meqvODxAAs651pUGt7bBRsVp3x24O1A== --osd-uuid 63d5774e-76b2-49c0-b726-79d5d68d6341 2024-06-27T21:53:42.606 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:42.603+0000 7f72aed8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:42.629 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:42.627+0000 7f72aed8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:42.631 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:42.627+0000 7f72aed8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:42.631 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:42.627+0000 7f72aed8b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:53:42.631 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:42.627+0000 7f72aed8b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:53:42.631 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:42.627+0000 7f72aed8b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:53:42.677 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:42.675+0000 7f72aed8b7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T21:53:49.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T21:53:49.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:53:49.138 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:53:49.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:53:49.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:53:49.717 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:53:49.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:53:49.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:53:49.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:53:49.719 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-06-27T21:53:49.719 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:53:49.752 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:49.751+0000 7f8d3d1777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:49.754 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:49.751+0000 7f8d3d1777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:49.756 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:49.755+0000 7f8d3d1777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:50.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:53:50.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:53:50.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:53:50.191 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:53:50.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:53:50.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:53:50.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:50.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:53:50.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:50.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:50.329 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:50.327+0000 7f8d3d1777c0 -1 Falling back to public interface 2024-06-27T21:53:50.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:51.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:51.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:51.684 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:53:51.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:53:51.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:51.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:52.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:52.944 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:52.942+0000 7f8d3d1777c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:53:53.124 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:53.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:53.125 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:53:53.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:53:53.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:53.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:53.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:54.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:54.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:54.657 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:53:54.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:53:54.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:54.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:55.130 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2327560811,v1:127.0.0.1:6803/2327560811] [v2:127.0.0.1:6804/2327560811,v1:127.0.0.1:6805/2327560811] exists,up 63d5774e-76b2-49c0-b726-79d5d68d6341 2024-06-27T21:53:55.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:53:55.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:53:55.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:53:55.131 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-06-27T21:53:55.131 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:53:55.131 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:53:55.132 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:53:55.132 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:55.132 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:55.132 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.0.asok 2024-06-27T21:53:55.132 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-06-27T21:53:55.132 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-06-27T21:53:55.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-06-27T21:53:55.280 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-06-27T21:53:55.280 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:53:55.280 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:53:55.280 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:53:55.280 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:55.280 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:55.280 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.0.asok 2024-06-27T21:53:55.280 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-06-27T21:53:55.280 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-06-27T21:53:55.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-06-27T21:53:55.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-06-27T21:53:55.381 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:53:55.381 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:53:55.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:53:55.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:53:55.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:53:55.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:53:55.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2024-06-27T21:53:56.157 INFO:tasks.workunit.client.0.smithi042.stderr:osd.0 is already down. 2024-06-27T21:53:56.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-06-27T21:53:56.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-06-27T21:53:56.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:53:56.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:53:56.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:53:56.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:56.174 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:53:56.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:53:56.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:56.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-06-27T21:53:56.620 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/2327560811,v1:127.0.0.1:6803/2327560811] [v2:127.0.0.1:6804/2327560811,v1:127.0.0.1:6805/2327560811] exists 63d5774e-76b2-49c0-b726-79d5d68d6341 2024-06-27T21:53:56.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:53:56.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:53:56.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:53:56.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-06-27T21:53:56.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-27T21:53:56.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:53:56.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-27T21:53:56.624 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:53:56.624 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:53:56.625 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:53:56.625 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:53:56.625 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:53:56.625 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:53:56.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:53:56.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:53:56.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:53:56.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:53:56.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:53:56.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:53:56.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:53:56.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:53:56.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-27T21:53:56.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-06-27T21:53:56.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:53:56.626 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:53:56.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-27T21:53:56.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-06-27T21:53:56.627 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-06-27T21:53:56.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-27T21:53:56.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-27T21:53:56.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-27T21:53:56.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-27T21:53:56.660 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:56.654+0000 7f0e989247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:56.662 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:56.658+0000 7f0e989247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:56.663 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:56.658+0000 7f0e989247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:53:57.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-27T21:53:57.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:53:57.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:53:57.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:53:57.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:53:57.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:57.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:53:57.071 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:53:57.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:57.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:57.237 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:53:57.234+0000 7f0e989247c0 -1 Falling back to public interface 2024-06-27T21:53:57.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:53:58.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:53:58.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:53:58.532 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:53:58.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:53:58.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:53:58.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:53:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:00.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:00.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:00.030 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:54:00.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:54:00.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:00.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:00.190 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:00.186+0000 7f0e989247c0 -1 osd.0 6 log_to_monitors true 2024-06-27T21:54:00.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:01.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:01.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:01.513 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:54:01.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:54:01.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:01.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:01.999 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/2713897688,v1:127.0.0.1:6803/2713897688] [v2:127.0.0.1:6804/2713897688,v1:127.0.0.1:6805/2713897688] exists,up 63d5774e-76b2-49c0-b726-79d5d68d6341 2024-06-27T21:54:01.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:54:01.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:54:01.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:54:02.000 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-06-27T21:54:02.000 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:54:02.000 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:54:02.001 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:54:02.001 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:02.001 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:02.001 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.0.asok 2024-06-27T21:54:02.001 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-06-27T21:54:02.001 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-06-27T21:54:02.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-06-27T21:54:02.126 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-06-27T21:54:02.127 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:54:02.127 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:54:02.127 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:54:02.127 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:02.127 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:02.128 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.0.asok 2024-06-27T21:54:02.128 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-06-27T21:54:02.128 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-06-27T21:54:02.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-06-27T21:54:02.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-06-27T21:54:02.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-06-27T21:54:02.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-06-27T21:54:02.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:54:02.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:54:02.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:54:02.250 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:02.250 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:02.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:02.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:02.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:02.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:02.358 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:54:02.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:54:02.359 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:54:02.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:54:02.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:54:02.360 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:54:02.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:02.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:54:02.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:54:02.362 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:02.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:54:02.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:54:02.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:54:02.366 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:54:02.367 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:02.367 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:02.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:54:02.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:54:02.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:54:02.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:54:02.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_destroy_osd td/ceph-helpers 2024-06-27T21:54:02.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2024-06-27T21:54:02.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2024-06-27T21:54:02.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:54:02.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:54:02.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:54:02.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:54:02.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:54:02.370 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:02.370 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:02.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:02.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:02.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:02.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:02.371 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:54:02.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:54:02.372 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:54:02.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:54:02.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:54:02.373 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:54:02.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:02.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:54:02.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:54:02.374 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:02.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:54:02.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:54:02.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:54:02.377 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:54:02.377 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:02.377 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:02.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:54:02.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:54:02.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:54:02.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:54:02.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:54:02.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:02.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:02.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:54:02.382 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:54:02.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:54:02.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:54:02.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:54:02.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2024-06-27T21:54:02.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:54:02.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:54:02.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:54:02.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:54:02.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:54:02.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-27T21:54:02.880 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:54:02.880 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:54:02.880 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:54:02.880 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:54:02.880 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:02.880 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:02.881 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:54:02.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:54:03.097 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:54:03.098 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:54:03.098 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:54:03.098 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:54:03.098 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:54:03.099 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:54:03.099 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:54:03.099 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:54:03.099 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:54:03.100 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:54:03.100 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:03.100 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:03.100 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:54:03.100 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:54:03.100 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:54:03.216 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:54:03.216 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:54:03.216 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:54:03.217 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:54:03.218 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:54:03.218 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:54:03.218 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:54:03.218 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:54:03.218 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:54:03.218 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:03.218 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:03.218 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:54:03.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:54:03.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:54:03.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2024-06-27T21:54:03.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:54:03.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:54:03.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:54:03.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:54:03.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:54:03.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:54:04.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:54:04.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:54:04.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:54:04.249 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:54:04.249 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:04.249 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:04.250 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:54:04.250 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:54:04.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:54:04.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2024-06-27T21:54:04.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T21:54:04.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:54:04.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:54:04.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:54:04.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:54:04.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:54:04.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:54:04.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:54:04.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:54:04.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:54:04.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:54:04.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:54:04.291 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:54:04.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62b8cb9c-e919-404f-b7b9-7e62a4759450 2024-06-27T21:54:04.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 62b8cb9c-e919-404f-b7b9-7e62a4759450' 2024-06-27T21:54:04.291 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 62b8cb9c-e919-404f-b7b9-7e62a4759450 2024-06-27T21:54:04.292 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:54:04.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8331mS8uqEhAApwqLrpdycixY3nkezh/g8A== 2024-06-27T21:54:04.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8331mS8uqEhAApwqLrpdycixY3nkezh/g8A=="}' 2024-06-27T21:54:04.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62b8cb9c-e919-404f-b7b9-7e62a4759450 -i td/ceph-helpers/0/new.json 2024-06-27T21:54:04.644 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:54:04.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T21:54:04.663 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8331mS8uqEhAApwqLrpdycixY3nkezh/g8A== --osd-uuid 62b8cb9c-e919-404f-b7b9-7e62a4759450 2024-06-27T21:54:04.704 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:04.702+0000 7ff9fbff87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:04.730 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:04.726+0000 7ff9fbff87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:04.732 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:04.730+0000 7ff9fbff87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:04.732 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:04.730+0000 7ff9fbff87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:54:04.732 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:04.730+0000 7ff9fbff87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:54:04.733 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:04.730+0000 7ff9fbff87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:54:04.778 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:04.774+0000 7ff9fbff87c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T21:54:11.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T21:54:11.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:54:11.271 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:54:11.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:54:11.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:54:11.894 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:54:11.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:54:11.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:54:11.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:54:11.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:54:11.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:54:11.927 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:11.922+0000 7fce3de3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:11.945 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:11.942+0000 7fce3de3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:11.946 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:11.942+0000 7fce3de3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:12.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:54:12.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:54:12.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:54:12.339 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:54:12.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:54:12.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:54:12.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:12.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:54:12.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:12.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:12.530 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:12.526+0000 7fce3de3b7c0 -1 Falling back to public interface 2024-06-27T21:54:12.767 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:13.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:13.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:13.769 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:54:13.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:54:13.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:13.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:14.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:15.176 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:15.170+0000 7fce3de3b7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:54:15.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:15.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:15.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:54:15.245 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:54:15.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:15.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:15.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:16.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:16.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:16.914 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:54:16.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:54:16.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:16.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:17.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:18.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:18.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:18.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:54:18.361 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:54:18.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:18.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:18.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:19.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:19.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:19.805 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:54:19.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:54:19.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:19.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:20.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:21.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:21.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:21.404 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:54:21.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:54:21.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:21.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:21.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:22.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:22.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:22.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:54:22.879 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:54:22.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:22.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:23.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:24.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:24.405 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:54:24.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:24.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:54:24.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:24.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:24.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:25.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:25.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:25.849 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:54:25.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:54:25.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:25.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:26.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:27.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:27.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:27.450 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:54:27.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:54:27.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:27.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:28.007 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:29.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:29.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:29.009 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:54:29.010 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:54:29.010 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:29.010 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:30.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:31.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:31.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:31.181 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:54:31.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:54:31.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:31.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:31.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:32.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:32.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:32.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:54:32.712 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:54:32.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:32.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:33.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:34.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:34.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:34.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:54:34.416 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:54:34.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:34.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:35.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:36.258 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:36.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:36.259 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:54:36.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:54:36.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:36.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:37.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:38.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:38.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:38.287 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:54:38.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:54:38.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:38.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:38.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:39.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:39.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:39.978 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:54:39.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:54:39.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:39.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:40.445 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1006233732,v1:127.0.0.1:6803/1006233732] [v2:127.0.0.1:6804/1006233732,v1:127.0.0.1:6805/1006233732] exists,up 62b8cb9c-e919-404f-b7b9-7e62a4759450 2024-06-27T21:54:40.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:54:40.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:54:40.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:54:40.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-06-27T21:54:40.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2024-06-27T21:54:40.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2024-06-27T21:54:40.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2024-06-27T21:54:40.996 INFO:tasks.workunit.client.0.smithi042.stderr:osd.0 is already out. 2024-06-27T21:54:41.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-06-27T21:54:41.013 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:41.013 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:41.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:41.014 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:41.014 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:41.321 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:41.321 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2024-06-27T21:54:41.831 INFO:tasks.workunit.client.0.smithi042.stderr:osd.0 is already down. 2024-06-27T21:54:41.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-06-27T21:54:42.359 INFO:tasks.workunit.client.0.smithi042.stderr:osd.0 does not exist 2024-06-27T21:54:42.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/ceph-helpers/0 2024-06-27T21:54:42.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers/0 2024-06-27T21:54:42.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:54:42.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers/0 KILL 2024-06-27T21:54:42.377 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:42.378 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:42.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:42.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:42.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:42.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:42.381 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:54:42.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:54:42.382 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:54:42.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:54:42.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:54:42.384 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:54:42.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:42.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:54:42.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:54:42.385 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:42.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:54:42.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:54:42.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers/0 2024-06-27T21:54:42.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:54:42.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:42.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:42.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:54:42.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:54:42.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:54:42.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/ceph-helpers/0 2024-06-27T21:54:42.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: ceph osd dump 2024-06-27T21:54:42.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: grep 'osd. ' 2024-06-27T21:54:42.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:804: test_destroy_osd: teardown td/ceph-helpers 2024-06-27T21:54:42.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:54:42.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:54:42.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:54:42.878 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:42.878 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:42.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:42.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:42.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:42.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:42.986 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:54:42.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:54:42.987 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:54:42.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:54:42.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:54:42.989 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:54:42.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:42.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:54:42.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:54:42.990 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:42.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:54:42.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:54:42.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:54:42.994 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:54:42.995 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:42.995 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:42.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:54:42.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:54:42.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:54:42.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:54:42.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_display_logs td/ceph-helpers 2024-06-27T21:54:42.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2118: test_display_logs: local dir=td/ceph-helpers 2024-06-27T21:54:42.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: test_display_logs: setup td/ceph-helpers 2024-06-27T21:54:42.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:54:42.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:54:42.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:54:42.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:54:42.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:54:42.998 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:42.998 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:42.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:42.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:42.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:42.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:42.999 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:54:42.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:54:42.999 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:54:43.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:54:43.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:54:43.000 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:54:43.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:43.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:54:43.002 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:54:43.002 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:43.002 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:54:43.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:54:43.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:54:43.005 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:54:43.005 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:43.005 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:43.005 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:54:43.005 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:54:43.006 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:54:43.006 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:54:43.007 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:54:43.007 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:43.007 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:43.007 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:54:43.009 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:54:43.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:54:43.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:54:43.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:54:43.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: test_display_logs: run_mon td/ceph-helpers a 2024-06-27T21:54:43.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:54:43.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:54:43.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:54:43.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:54:43.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:54:43.010 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-27T21:54:43.483 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:54:43.484 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:54:43.484 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:54:43.484 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:54:43.484 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:43.484 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:43.485 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:54:43.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:54:43.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:54:43.710 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:54:43.710 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:54:43.710 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:54:43.710 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:54:43.711 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:54:43.712 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:54:43.712 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:54:43.712 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:54:43.712 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:54:43.712 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:43.712 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:43.712 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:54:43.712 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:54:43.713 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:54:43.827 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:54:43.827 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:54:43.827 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:54:43.827 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:54:43.828 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:54:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:54:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:54:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:54:43.829 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:54:43.829 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:43.829 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:43.829 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:54:43.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:54:43.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:54:43.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: test_display_logs: kill_daemons td/ceph-helpers 2024-06-27T21:54:43.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:43.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:43.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:43.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:43.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:45.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:45.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: test_display_logs: display_logs td/ceph-helpers 2024-06-27T21:54:45.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: display_logs: local dir=td/ceph-helpers 2024-06-27T21:54:45.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2024-06-27T21:54:45.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: display_logs: read file 2024-06-27T21:54:45.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2024-06-27T21:54:45.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: display_logs: cat td/ceph-helpers/mon.a.log 2024-06-27T21:54:45.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: display_logs: read file 2024-06-27T21:54:45.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2024-06-27T21:54:45.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: test_display_logs: teardown td/ceph-helpers 2024-06-27T21:54:45.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:54:45.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:54:45.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:54:45.266 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:45.266 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:45.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:45.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:45.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:45.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:45.270 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:54:45.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:54:45.272 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:54:45.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:54:45.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:54:45.273 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:54:45.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:45.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:54:45.275 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:54:45.275 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:45.276 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:54:45.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:54:45.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:54:45.280 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:54:45.280 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:45.280 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:45.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:54:45.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:54:45.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:54:45.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:54:45.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2024-06-27T21:54:45.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2024-06-27T21:54:45.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2091: test_erasure_code_plugin_exists: setup td/ceph-helpers 2024-06-27T21:54:45.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:54:45.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:54:45.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:54:45.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:54:45.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:54:45.284 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:45.284 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:45.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:45.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:45.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:45.285 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:45.286 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:54:45.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:54:45.287 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:54:45.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:54:45.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:54:45.289 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:54:45.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:45.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:54:45.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:54:45.291 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:45.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:54:45.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:54:45.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:54:45.295 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:54:45.295 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:45.295 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:45.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:54:45.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:54:45.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:54:45.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:54:45.298 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:54:45.298 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:45.298 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:45.298 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:54:45.301 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:54:45.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:54:45.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:54:45.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:54:45.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2092: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2024-06-27T21:54:45.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:54:45.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:54:45.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:54:45.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:54:45.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:54:45.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-27T21:54:45.773 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:54:45.773 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:54:45.773 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:54:45.774 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:54:45.774 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:45.774 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:45.775 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:54:45.775 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:54:45.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:54:45.986 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:54:45.986 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:54:45.986 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:54:45.986 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:54:45.987 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:54:45.987 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:54:45.987 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:54:45.988 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:54:45.988 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:54:45.988 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:45.988 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:45.988 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:54:45.988 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:54:45.988 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:54:46.107 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:54:46.107 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:54:46.107 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:54:46.107 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:54:46.107 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:54:46.108 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:54:46.108 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:54:46.108 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:54:46.108 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:54:46.109 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:46.109 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:46.109 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:54:46.109 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:54:46.109 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:54:46.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2093: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2024-06-27T21:54:46.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:54:46.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:54:46.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:54:46.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:54:46.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:54:46.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:54:47.160 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:54:47.160 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:54:47.161 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:54:47.161 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:54:47.161 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:47.161 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:47.161 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:54:47.162 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:54:47.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:54:47.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2094: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2024-06-27T21:54:47.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: erasure_code_plugin_exists: local plugin=jerasure 2024-06-27T21:54:47.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: local status 2024-06-27T21:54:47.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local grepstr 2024-06-27T21:54:47.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: local s 2024-06-27T21:54:47.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: case `uname` in 2024-06-27T21:54:47.242 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: uname 2024-06-27T21:54:47.243 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2024-06-27T21:54:47.243 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2024-06-27T21:54:47.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: erasure_code_plugin_exists: s= 2024-06-27T21:54:47.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: local status=0 2024-06-27T21:54:47.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-06-27T21:54:47.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-06-27T21:54:48.063 INFO:tasks.workunit.client.0.smithi042.stderr:erasure-code-profile TESTPROFILE does not exist 2024-06-27T21:54:48.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: erasure_code_plugin_exists: return 0 2024-06-27T21:54:48.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2024-06-27T21:54:48.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: erasure_code_plugin_exists: local plugin=FAKE 2024-06-27T21:54:48.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: local status 2024-06-27T21:54:48.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local grepstr 2024-06-27T21:54:48.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: local s 2024-06-27T21:54:48.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: case `uname` in 2024-06-27T21:54:48.080 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: uname 2024-06-27T21:54:48.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2024-06-27T21:54:48.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2024-06-27T21:54:48.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so): /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2024-06-27T21:54:48.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: local status=5 2024-06-27T21:54:48.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2024-06-27T21:54:48.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so):' /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2024-06-27T21:54:48.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2024-06-27T21:54:48.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: erasure_code_plugin_exists: return 5 2024-06-27T21:54:48.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2024-06-27T21:54:48.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:54:48.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:54:48.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:54:48.297 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:48.297 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:48.298 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:48.298 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:48.298 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:48.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:48.404 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:54:48.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:54:48.405 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:54:48.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:54:48.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:54:48.407 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:54:48.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:48.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:54:48.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:54:48.409 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:48.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:54:48.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:54:48.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:54:48.412 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:54:48.413 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:48.413 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:48.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:54:48.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:54:48.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:54:48.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:54:48.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_expect_failure td/ceph-helpers 2024-06-27T21:54:48.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1999: test_expect_failure: local dir=td/ceph-helpers 2024-06-27T21:54:48.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: test_expect_failure: setup td/ceph-helpers 2024-06-27T21:54:48.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:54:48.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:54:48.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:54:48.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:54:48.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:54:48.416 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:48.416 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:48.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:48.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:48.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:48.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:48.417 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:54:48.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:54:48.418 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:54:48.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:54:48.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:54:48.419 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:54:48.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:48.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:54:48.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:54:48.420 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:48.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:54:48.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:54:48.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:54:48.424 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:54:48.424 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:48.424 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:48.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:54:48.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:54:48.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:54:48.425 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:54:48.426 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:54:48.426 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:48.426 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:48.426 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:54:48.428 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:54:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:54:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:54:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:54:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2024-06-27T21:54:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/ceph-helpers 2024-06-27T21:54:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-06-27T21:54:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=FAIL 2024-06-27T21:54:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-06-27T21:54:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-06-27T21:54:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: bash -c 'echo FAIL ; exit 1' 2024-06-27T21:54:48.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-06-27T21:54:48.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-06-27T21:54:48.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-06-27T21:54:48.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-06-27T21:54:48.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2004: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2024-06-27T21:54:48.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/ceph-helpers 2024-06-27T21:54:48.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-06-27T21:54:48.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=FAIL 2024-06-27T21:54:48.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-06-27T21:54:48.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-06-27T21:54:48.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: bash -c 'echo FAIL ; exit 0' 2024-06-27T21:54:48.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: success=true 2024-06-27T21:54:48.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: true 2024-06-27T21:54:48.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: cat td/ceph-helpers/out 2024-06-27T21:54:48.433 INFO:tasks.workunit.client.0.smithi042.stderr:FAIL 2024-06-27T21:54:48.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: expect_failure: return 1 2024-06-27T21:54:48.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-06-27T21:54:48.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2007: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2024-06-27T21:54:48.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/ceph-helpers 2024-06-27T21:54:48.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-06-27T21:54:48.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=FAIL 2024-06-27T21:54:48.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-06-27T21:54:48.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-06-27T21:54:48.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2024-06-27T21:54:48.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-06-27T21:54:48.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-06-27T21:54:48.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-06-27T21:54:48.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: cat td/ceph-helpers/out 2024-06-27T21:54:48.437 INFO:tasks.workunit.client.0.smithi042.stderr:UNEXPECTED 2024-06-27T21:54:48.437 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: expect_failure: return 1 2024-06-27T21:54:48.437 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-06-27T21:54:48.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: test_expect_failure: teardown td/ceph-helpers 2024-06-27T21:54:48.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:54:48.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:54:48.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:54:48.439 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:48.439 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:48.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:48.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:48.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:48.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:48.441 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:54:48.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:54:48.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:54:48.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:54:48.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:54:48.444 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:54:48.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:48.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:54:48.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:54:48.445 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:48.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:54:48.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:54:48.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:54:48.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:54:48.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:48.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:48.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:54:48.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:54:48.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:54:48.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:54:48.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_flush_pg_stats td/ceph-helpers 2024-06-27T21:54:48.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: test_flush_pg_stats: local dir=td/ceph-helpers 2024-06-27T21:54:48.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: test_flush_pg_stats: setup td/ceph-helpers 2024-06-27T21:54:48.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:54:48.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:54:48.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:54:48.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:54:48.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:54:48.452 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:54:48.452 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:54:48.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:54:48.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:54:48.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:54:48.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:54:48.453 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:54:48.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:54:48.454 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:54:48.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:54:48.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:54:48.455 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:54:48.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:48.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:54:48.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:54:48.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:54:48.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:54:48.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:54:48.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:54:48.460 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:54:48.460 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:48.460 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:48.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:54:48.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:54:48.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:54:48.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:54:48.463 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:54:48.463 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:48.463 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:48.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:54:48.465 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:54:48.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:54:48.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:54:48.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:54:48.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:54:48.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:54:48.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:54:48.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:54:48.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:54:48.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:54:48.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:54:48.917 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:54:48.917 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:54:48.917 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:54:48.918 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:54:48.918 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:48.918 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:48.918 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:54:48.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:54:49.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:54:49.144 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:54:49.144 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:54:49.144 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:54:49.145 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:54:49.146 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:54:49.146 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:54:49.146 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:54:49.146 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:54:49.146 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:54:49.146 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:49.146 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:49.146 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:54:49.147 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:54:49.147 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:54:49.261 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:54:49.261 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:54:49.261 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:54:49.261 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:54:49.262 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:54:49.263 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:54:49.263 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:54:49.263 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:54:49.263 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:54:49.263 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:49.263 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:49.263 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:54:49.263 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:54:49.264 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:54:49.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: test_flush_pg_stats: run_mgr td/ceph-helpers x 2024-06-27T21:54:49.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:54:49.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:54:49.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:54:49.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:54:49.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:54:49.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:54:50.295 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:54:50.295 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:54:50.295 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:54:50.295 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:54:50.296 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:50.296 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:50.296 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:54:50.296 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:54:50.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:54:50.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: test_flush_pg_stats: run_osd td/ceph-helpers 0 2024-06-27T21:54:50.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T21:54:50.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:54:50.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:54:50.362 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:54:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:54:50.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:54:50.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:54:50.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:54:50.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:54:50.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:54:50.364 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:54:50.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae5609cd-6455-447b-aadc-263062587367 2024-06-27T21:54:50.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ae5609cd-6455-447b-aadc-263062587367' 2024-06-27T21:54:50.365 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 ae5609cd-6455-447b-aadc-263062587367 2024-06-27T21:54:50.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:54:50.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCq331mXC/EFhAAvKECcyZwvg4tV1defFY+oQ== 2024-06-27T21:54:50.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCq331mXC/EFhAAvKECcyZwvg4tV1defFY+oQ=="}' 2024-06-27T21:54:50.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae5609cd-6455-447b-aadc-263062587367 -i td/ceph-helpers/0/new.json 2024-06-27T21:54:50.684 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:54:50.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T21:54:50.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCq331mXC/EFhAAvKECcyZwvg4tV1defFY+oQ== --osd-uuid ae5609cd-6455-447b-aadc-263062587367 2024-06-27T21:54:50.743 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:50.738+0000 7f48fe3c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:50.746 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:50.742+0000 7f48fe3c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:50.747 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:50.742+0000 7f48fe3c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:50.747 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:50.742+0000 7f48fe3c57c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:54:50.748 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:50.742+0000 7f48fe3c57c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:54:50.748 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:50.742+0000 7f48fe3c57c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:54:50.790 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:50.786+0000 7f48fe3c57c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T21:54:57.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T21:54:57.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:54:57.231 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:54:57.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:54:57.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:54:57.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:54:57.814 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:54:57.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:54:57.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:54:57.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:54:57.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:54:57.846 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:57.842+0000 7f44214347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:57.848 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:57.846+0000 7f44214347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:57.850 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:57.846+0000 7f44214347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:54:58.276 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:54:58.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:54:58.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:54:58.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:54:58.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:54:58.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:58.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:54:58.277 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:54:58.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:58.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:54:58.421 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:54:58.418+0000 7f44214347c0 -1 Falling back to public interface 2024-06-27T21:54:58.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:54:59.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:54:59.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:54:59.726 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:54:59.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:54:59.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:54:59.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:00.158 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:01.004 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:01.002+0000 7f44214347c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:55:01.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:01.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:01.160 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:55:01.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:55:01.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:01.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:01.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:02.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:02.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:02.788 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:55:02.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:55:02.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:02.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:03.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:04.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:04.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:04.255 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:55:04.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:55:04.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:04.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:04.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:05.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:05.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:05.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:55:05.704 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:55:05.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:05.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:06.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:07.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:07.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:07.220 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:55:07.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:55:07.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:07.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:07.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:08.696 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:08.696 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:08.696 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:55:08.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:55:08.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:08.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:09.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:10.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:10.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:10.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:55:10.208 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:55:10.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:10.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:10.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:11.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:11.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:11.659 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:55:11.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:55:11.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:11.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:12.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:13.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:13.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:13.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:55:13.213 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:55:13.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:13.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:13.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:14.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:14.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:14.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:55:14.681 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:55:14.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:14.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:15.502 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:16.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:16.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:16.504 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:55:16.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:55:16.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:16.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:17.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:18.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:18.318 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:18.318 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:55:18.319 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:55:18.319 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:18.319 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:19.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:20.106 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:20.106 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:20.107 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:55:20.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:55:20.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:20.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:21.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:22.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:22.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:22.206 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:55:22.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:55:22.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:22.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:23.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:24.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:24.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:24.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:55:24.150 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:55:24.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:24.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:25.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:26.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:26.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:26.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:55:26.051 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:55:26.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:26.052 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:26.552 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4247993976,v1:127.0.0.1:6803/4247993976] [v2:127.0.0.1:6804/4247993976,v1:127.0.0.1:6805/4247993976] exists,up ae5609cd-6455-447b-aadc-263062587367 2024-06-27T21:55:26.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:55:26.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:55:26.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:55:26.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: test_flush_pg_stats: create_rbd_pool 2024-06-27T21:55:26.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T21:55:27.019 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T21:55:27.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T21:55:27.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T21:55:27.716 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T21:55:27.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T21:55:28.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T21:55:29.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: test_flush_pg_stats: rados -p rbd put obj /etc/group 2024-06-27T21:55:30.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: test_flush_pg_stats: flush_pg_stats 2024-06-27T21:55:30.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:55:30.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:55:30.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T21:55:30.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:55:30.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:55:30.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:55:30.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836487 2024-06-27T21:55:30.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836487 2024-06-27T21:55:30.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487' 2024-06-27T21:55:30.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:55:30.735 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836487 2024-06-27T21:55:30.735 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:55:30.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:55:30.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-06-27T21:55:30.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:55:30.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836487 2024-06-27T21:55:30.737 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836487 2024-06-27T21:55:30.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-06-27T21:55:30.738 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:55:31.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836487 2024-06-27T21:55:31.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T21:55:32.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T21:55:32.153 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:55:32.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836487 2024-06-27T21:55:32.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2024-06-27T21:55:32.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2024-06-27T21:55:32.607 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: test_flush_pg_stats: ceph df detail --format=json 2024-06-27T21:55:33.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: test_flush_pg_stats: stored=6543 2024-06-27T21:55:33.180 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: test_flush_pg_stats: ceph df detail --format=json 2024-06-27T21:55:33.180 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2024-06-27T21:55:33.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: test_flush_pg_stats: stored_raw=6543 2024-06-27T21:55:33.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: test_flush_pg_stats: test 6543 -gt 0 2024-06-27T21:55:33.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: test_flush_pg_stats: test 6543 == 6543 2024-06-27T21:55:33.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: test_flush_pg_stats: teardown td/ceph-helpers 2024-06-27T21:55:33.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:55:33.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:55:33.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:55:33.760 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:55:33.761 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:55:33.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:55:33.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:55:33.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:55:34.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:55:34.070 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:55:34.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:55:34.072 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:55:34.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:55:34.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:55:34.074 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:55:34.075 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:55:34.075 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:55:34.075 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:55:34.076 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:55:34.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:55:34.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:55:34.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:55:34.086 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:55:34.086 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:55:34.086 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:55:34.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:55:34.089 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:55:34.089 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:55:34.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:55:34.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_config td/ceph-helpers 2024-06-27T21:55:34.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: local dir=td/ceph-helpers 2024-06-27T21:55:34.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: test_get_config: setup td/ceph-helpers 2024-06-27T21:55:34.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:55:34.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:55:34.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:55:34.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:55:34.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:55:34.091 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:55:34.091 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:55:34.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:55:34.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:55:34.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:55:34.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:55:34.093 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:55:34.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:55:34.093 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:55:34.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:55:34.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:55:34.094 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:55:34.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:55:34.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:55:34.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:55:34.096 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:55:34.097 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:55:34.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:55:34.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:55:34.100 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:55:34.100 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:55:34.100 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:55:34.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:55:34.101 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:55:34.101 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:55:34.101 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:55:34.102 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:55:34.102 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:55:34.102 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:55:34.102 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:55:34.103 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:55:34.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:55:34.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:55:34.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:55:34.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1140: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:55:34.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:55:34.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:55:34.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:55:34.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:55:34.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:55:34.106 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:55:34.589 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:55:34.589 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:55:34.589 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:55:34.591 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:55:34.591 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:55:34.591 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:55:34.591 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:55:34.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:55:34.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:55:34.828 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:55:34.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:55:34.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:55:34.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:55:34.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:55:34.830 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:55:34.830 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:55:34.830 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:55:34.830 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:55:34.830 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:55:34.830 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:55:34.830 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:55:34.830 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:55:34.831 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:55:34.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:55:34.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:55:34.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:55:34.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:55:34.953 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:55:34.953 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:55:34.953 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:55:34.953 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:55:34.953 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:55:34.953 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:55:34.953 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:55:34.953 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:55:34.954 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:55:34.954 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:55:35.067 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: get_config mon a osd_pool_default_size 2024-06-27T21:55:35.067 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:55:35.067 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:55:35.067 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_pool_default_size 2024-06-27T21:55:35.067 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_pool_default_size 2024-06-27T21:55:35.068 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:55:35.068 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:55:35.068 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:55:35.069 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:55:35.069 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:55:35.069 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:55:35.069 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:55:35.069 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:55:35.069 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get osd_pool_default_size 2024-06-27T21:55:35.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: test 1 = 1 2024-06-27T21:55:35.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: test_get_config: run_mgr td/ceph-helpers x 2024-06-27T21:55:35.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:55:35.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:55:35.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:55:35.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:55:35.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:55:35.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:55:35.693 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:55:35.693 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:55:35.693 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:55:35.694 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:55:35.694 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:55:35.694 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:55:35.694 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:55:35.695 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:55:35.696 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:55:35.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1143: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2024-06-27T21:55:35.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T21:55:35.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:55:35.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:55:35.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:55:35.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:55:35.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:55:35.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:55:35.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:55:35.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:55:35.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:55:35.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:55:35.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:55:35.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:55:35.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:55:35.779 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:55:35.779 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:55:35.779 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:55:35.779 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:55:35.779 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:55:35.779 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:55:35.779 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:55:35.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:55:35.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:55:35.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:55:35.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:55:35.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:55:35.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:55:35.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:55:35.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:55:35.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:55:35.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:55:35.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:55:35.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_max_scrubs=3 2024-06-27T21:55:35.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:55:35.781 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:55:35.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5267bdce-3ad8-4a27-a140-0f08af0db652 2024-06-27T21:55:35.781 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 5267bdce-3ad8-4a27-a140-0f08af0db652 2024-06-27T21:55:35.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5267bdce-3ad8-4a27-a140-0f08af0db652' 2024-06-27T21:55:35.782 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:55:35.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDX331mVX9yLxAASPJIx1iQvKz2ZWvovpVDBQ== 2024-06-27T21:55:35.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDX331mVX9yLxAASPJIx1iQvKz2ZWvovpVDBQ=="}' 2024-06-27T21:55:35.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5267bdce-3ad8-4a27-a140-0f08af0db652 -i td/ceph-helpers/0/new.json 2024-06-27T21:55:36.113 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:55:36.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T21:55:36.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQDX331mVX9yLxAASPJIx1iQvKz2ZWvovpVDBQ== --osd-uuid 5267bdce-3ad8-4a27-a140-0f08af0db652 2024-06-27T21:55:36.162 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:36.158+0000 7ff83a2b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:55:36.164 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:36.162+0000 7ff83a2b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:55:36.166 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:36.162+0000 7ff83a2b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:55:36.166 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:36.162+0000 7ff83a2b87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:55:36.167 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:36.162+0000 7ff83a2b87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:55:36.167 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:36.162+0000 7ff83a2b87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:55:36.222 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:36.218+0000 7ff83a2b87c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T21:55:42.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T21:55:42.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:55:42.722 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:55:42.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:55:42.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:55:43.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:55:43.363 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:55:43.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2024-06-27T21:55:43.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:55:43.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:55:43.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:55:43.396 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:43.394+0000 7f800a95b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:55:43.415 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:43.410+0000 7f800a95b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:55:43.416 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:43.414+0000 7f800a95b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:55:43.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:55:43.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:55:43.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:55:43.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:55:43.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:55:43.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:43.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:55:43.802 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:55:43.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:43.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:43.989 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:43.986+0000 7f800a95b7c0 -1 Falling back to public interface 2024-06-27T21:55:44.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:45.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:45.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:45.238 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:55:45.238 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:55:45.238 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:45.238 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:45.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:46.594 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:55:46.590+0000 7f800a95b7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:55:46.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:46.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:46.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:55:46.713 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:55:46.714 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:46.714 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:47.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:48.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:48.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:48.324 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:55:48.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:55:48.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:48.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:48.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:49.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:49.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:49.781 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:55:49.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:55:49.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:49.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:50.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:51.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:51.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:51.280 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:55:51.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:55:51.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:51.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:51.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:52.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:52.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:52.791 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:55:52.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:55:52.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:52.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:53.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:54.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:54.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:54.252 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:55:54.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:55:54.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:54.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:54.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:55.839 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:55.839 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:55.840 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:55:55.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:55:55.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:55.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:56.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:57.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:57.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:57.435 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:55:57.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:55:57.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:57.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:57.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:55:58.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:55:58.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:55:58.881 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:55:58.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:55:58.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:55:58.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:55:59.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:00.499 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:00.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:00.500 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:56:00.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:56:00.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:00.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:01.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:02.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:02.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:02.348 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:56:02.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:56:02.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:02.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:02.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:03.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:03.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:03.834 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:56:03.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:56:03.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:03.835 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:04.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:05.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:05.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:05.633 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:56:05.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:56:05.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:05.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:06.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:07.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:07.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:07.702 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:56:07.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:56:07.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:07.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:08.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:09.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:09.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:09.612 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:56:09.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:56:09.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:09.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:10.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:11.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:11.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:11.466 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:56:11.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:56:11.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:11.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:11.901 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3465876031,v1:127.0.0.1:6803/3465876031] [v2:127.0.0.1:6804/3465876031,v1:127.0.0.1:6805/3465876031] exists,up 5267bdce-3ad8-4a27-a140-0f08af0db652 2024-06-27T21:56:11.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:56:11.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:56:11.902 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:56:11.902 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: get_config osd 0 osd_max_scrubs 2024-06-27T21:56:11.902 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-06-27T21:56:11.902 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-06-27T21:56:11.902 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_max_scrubs 2024-06-27T21:56:11.903 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_max_scrubs 2024-06-27T21:56:11.903 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-06-27T21:56:11.903 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T21:56:11.903 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T21:56:11.904 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:56:11.904 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:11.904 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:11.905 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.0.asok 2024-06-27T21:56:11.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:56:11.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-osd.0.asok config get osd_max_scrubs 2024-06-27T21:56:12.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: test 3 = 3 2024-06-27T21:56:12.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1145: test_get_config: teardown td/ceph-helpers 2024-06-27T21:56:12.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:56:12.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:56:12.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:56:12.029 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:56:12.030 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:56:12.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:56:12.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:56:12.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:56:12.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:56:12.138 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:56:12.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:56:12.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:56:12.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:56:12.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:56:12.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:56:12.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:56:12.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:56:12.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:56:12.142 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:56:12.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:56:12.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:56:12.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:56:12.149 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:56:12.149 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:12.150 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:12.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: test_get_is_making_recovery_progress: setup td/ceph-helpers 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:56:12.152 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:56:12.153 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:56:12.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:56:12.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:56:12.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:56:12.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:56:12.153 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:56:12.154 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:56:12.154 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:56:12.155 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:56:12.155 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:56:12.156 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:56:12.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:56:12.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:56:12.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:56:12.157 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:56:12.158 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:56:12.158 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:56:12.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:56:12.160 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:56:12.160 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:12.160 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:12.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:56:12.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:56:12.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:56:12.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:56:12.162 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:56:12.162 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:12.162 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:12.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:56:12.164 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:56:12.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:56:12.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:56:12.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:56:12.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1347: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2024-06-27T21:56:12.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:56:12.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:56:12.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:56:12.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:56:12.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:56:12.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-27T21:56:12.624 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:56:12.625 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:56:12.625 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:56:12.625 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:56:12.625 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:12.625 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:12.625 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:56:12.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T21:56:12.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:56:12.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:56:12.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:56:12.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:56:12.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:56:12.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:56:12.856 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:56:12.856 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:56:12.856 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:56:12.857 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:56:12.857 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:12.857 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:12.857 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:56:12.857 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:56:12.857 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:56:12.972 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:56:12.972 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:56:12.972 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:56:12.972 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:56:12.972 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:56:12.973 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:56:12.974 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:56:12.974 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:56:12.974 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:56:12.974 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:12.974 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:12.974 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:56:12.974 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:56:12.974 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:56:13.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1348: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2024-06-27T21:56:13.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:56:13.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:56:13.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:56:13.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:56:13.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:56:13.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:56:14.003 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:56:14.003 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:56:14.003 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:56:14.004 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:56:14.004 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:14.004 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:14.004 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:56:14.005 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:56:14.006 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:56:14.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1349: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2024-06-27T21:56:14.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-27T21:56:14.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-27T21:56:14.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-27T21:56:14.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-27T21:56:14.040 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-27T21:56:14.040 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-27T21:56:14.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-27T21:56:14.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-27T21:56:14.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1350: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2024-06-27T21:56:14.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:56:14.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:56:14.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:56:14.277 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:56:14.278 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:56:14.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:56:14.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:56:14.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:56:14.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:56:14.384 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:56:14.384 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:56:14.385 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:56:14.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:56:14.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:56:14.386 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:56:14.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:56:14.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:56:14.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:56:14.388 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:56:14.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:56:14.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:56:14.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:56:14.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:56:14.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:14.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:14.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:56:14.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:56:14.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:56:14.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:56:14.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2024-06-27T21:56:14.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2024-06-27T21:56:14.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1535: test_get_last_scrub_stamp: setup td/ceph-helpers 2024-06-27T21:56:14.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:56:14.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:56:14.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:56:14.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:56:14.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:56:14.395 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:56:14.395 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:56:14.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:56:14.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:56:14.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:56:14.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:56:14.397 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:56:14.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:56:14.398 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:56:14.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:56:14.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:56:14.400 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:56:14.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:56:14.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:56:14.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:56:14.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:56:14.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:56:14.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:56:14.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:56:14.404 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:56:14.404 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:14.404 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:14.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:56:14.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:56:14.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:56:14.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:56:14.406 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:56:14.406 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:14.406 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:14.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:56:14.408 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:56:14.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:56:14.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:56:14.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:56:14.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1536: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:56:14.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:56:14.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:56:14.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:56:14.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:56:14.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:56:14.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:56:14.891 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:56:14.891 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:56:14.891 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:56:14.892 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:56:14.892 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:14.892 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:14.892 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:56:14.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:56:15.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:56:15.127 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:56:15.127 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:56:15.127 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:56:15.127 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:56:15.127 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:56:15.128 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:56:15.128 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:56:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:56:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:56:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:56:15.129 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:56:15.129 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:56:15.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:56:15.248 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:56:15.248 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:56:15.248 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:56:15.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:56:15.249 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:56:15.249 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:56:15.249 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:56:15.249 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:56:15.249 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:15.249 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:15.249 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:56:15.250 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:56:15.250 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:56:15.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1537: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2024-06-27T21:56:15.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:56:15.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:56:15.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:56:15.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:56:15.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:56:15.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:56:16.275 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:56:16.276 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:56:16.276 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:56:16.276 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:56:16.276 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:16.276 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:16.276 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:56:16.276 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:56:16.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:56:16.309 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1538: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2024-06-27T21:56:16.309 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T21:56:16.309 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:56:16.309 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:56:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:56:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:56:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:56:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:56:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:56:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:56:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:56:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:56:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:56:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:56:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:56:16.311 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:56:16.311 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:56:16.311 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:56:16.311 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:56:16.311 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:56:16.311 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:56:16.311 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:56:16.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:56:16.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:56:16.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:56:16.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:56:16.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:56:16.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:56:16.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:56:16.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:56:16.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:56:16.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:56:16.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:56:16.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:56:16.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:56:16.312 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:56:16.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe798fa9-dd6c-4e25-bfd1-0734c264f202 2024-06-27T21:56:16.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fe798fa9-dd6c-4e25-bfd1-0734c264f202' 2024-06-27T21:56:16.313 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 fe798fa9-dd6c-4e25-bfd1-0734c264f202 2024-06-27T21:56:16.313 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:56:16.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAA4H1m0viOExAAl+j7M/U5l81SAiEOMZzjXg== 2024-06-27T21:56:16.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAA4H1m0viOExAAl+j7M/U5l81SAiEOMZzjXg=="}' 2024-06-27T21:56:16.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe798fa9-dd6c-4e25-bfd1-0734c264f202 -i td/ceph-helpers/0/new.json 2024-06-27T21:56:16.648 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:56:16.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T21:56:16.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAA4H1m0viOExAAl+j7M/U5l81SAiEOMZzjXg== --osd-uuid fe798fa9-dd6c-4e25-bfd1-0734c264f202 2024-06-27T21:56:16.700 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:16.697+0000 7f89052657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:56:16.733 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:16.729+0000 7f89052657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:56:16.735 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:16.729+0000 7f89052657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:56:16.735 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:16.729+0000 7f89052657c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:56:16.735 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:16.729+0000 7f89052657c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:56:16.735 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:16.729+0000 7f89052657c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:56:16.773 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:16.769+0000 7f89052657c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T21:56:23.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T21:56:23.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:56:23.270 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:56:23.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:56:23.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:56:23.856 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:56:23.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:56:23.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:56:23.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:56:23.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:56:23.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:56:23.890 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:23.885+0000 7f73867607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:56:23.916 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:23.913+0000 7f73867607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:56:23.918 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:23.913+0000 7f73867607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:56:24.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:56:24.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:56:24.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:56:24.350 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:56:24.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:56:24.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:56:24.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:24.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:56:24.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:24.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:24.481 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:24.477+0000 7f73867607c0 -1 Falling back to public interface 2024-06-27T21:56:24.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:25.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:25.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:25.841 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:56:25.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:56:25.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:25.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:26.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:27.171 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:56:27.165+0000 7f73867607c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:56:27.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:27.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:27.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:56:27.273 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:56:27.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:27.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:27.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:28.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:28.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:28.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:56:28.847 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:56:28.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:28.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:29.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:30.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:30.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:30.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:56:30.301 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:56:30.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:30.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:30.753 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:31.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:31.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:31.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:56:31.755 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:56:31.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:31.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:32.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:33.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:33.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:33.289 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:56:33.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:56:33.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:33.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:33.733 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:34.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:34.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:34.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:56:34.735 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:56:34.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:34.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:35.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:36.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:36.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:36.275 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:56:36.275 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:56:36.275 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:36.275 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:36.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:37.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:37.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:37.723 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:56:37.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:56:37.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:37.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:38.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:39.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:39.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:39.270 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:56:39.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:56:39.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:39.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:39.720 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:40.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:40.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:40.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:56:40.723 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:56:40.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:40.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:41.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:42.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:42.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:42.759 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:56:42.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:56:42.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:42.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:43.589 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:44.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:44.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:44.591 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:56:44.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:56:44.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:44.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:45.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:46.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:46.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:46.521 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:56:46.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:56:46.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:46.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:47.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:48.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:48.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:48.339 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:56:48.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:56:48.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:48.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:49.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:50.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:50.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:50.087 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:56:50.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:56:50.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:50.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:50.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:56:51.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:56:51.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:56:51.958 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:56:51.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:56:51.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:56:51.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:56:52.417 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/468935565,v1:127.0.0.1:6803/468935565] [v2:127.0.0.1:6804/468935565,v1:127.0.0.1:6805/468935565] exists,up fe798fa9-dd6c-4e25-bfd1-0734c264f202 2024-06-27T21:56:52.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:56:52.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:56:52.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:56:52.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1539: test_get_last_scrub_stamp: create_rbd_pool 2024-06-27T21:56:52.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T21:56:52.830 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T21:56:52.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T21:56:52.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T21:56:53.411 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T21:56:53.427 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T21:56:54.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T21:56:55.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1540: test_get_last_scrub_stamp: wait_for_clean 2024-06-27T21:56:55.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T21:56:55.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T21:56:55.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T21:56:55.605 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T21:56:55.606 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T21:56:55.606 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T21:56:55.607 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T21:56:55.607 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T21:56:55.607 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T21:56:55.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T21:56:55.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T21:56:55.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T21:56:55.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T21:56:55.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:56:55.732 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:56:56.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T21:56:56.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:56:56.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:56:56.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:56:56.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836489 2024-06-27T21:56:56.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836489 2024-06-27T21:56:56.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489' 2024-06-27T21:56:56.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:56:56.396 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836489 2024-06-27T21:56:56.396 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:56:56.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:56:56.397 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836489 2024-06-27T21:56:56.398 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:56:56.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836489 2024-06-27T21:56:56.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-06-27T21:56:56.399 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836489 2024-06-27T21:56:56.399 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:56:56.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836489 2024-06-27T21:56:56.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T21:56:57.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T21:56:57.881 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:56:58.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836490 -lt 21474836489 2024-06-27T21:56:58.335 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T21:56:58.335 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:56:58.335 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:56:58.907 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T21:56:58.907 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T21:56:58.908 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T21:56:58.908 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T21:56:58.908 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T21:56:58.908 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T21:56:58.908 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T21:56:58.909 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T21:56:59.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T21:56:59.358 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T21:56:59.358 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:56:59.358 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:56:59.930 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T21:56:59.930 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T21:56:59.930 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T21:56:59.931 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2024-06-27T21:56:59.931 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T21:56:59.931 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T21:56:59.931 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T21:56:59.931 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T21:57:00.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: stamp=2024-06-27T21:56:53.234178+0000 2024-06-27T21:57:00.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1542: test_get_last_scrub_stamp: test -n 2024-06-27T21:56:53.234178+0000 2024-06-27T21:57:00.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1543: test_get_last_scrub_stamp: teardown td/ceph-helpers 2024-06-27T21:57:00.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:57:00.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:57:00.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:57:00.349 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:57:00.349 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:57:00.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:57:00.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:57:00.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:57:00.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:57:00.658 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:57:00.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:57:00.659 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:57:00.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:57:00.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:57:00.662 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:57:00.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:57:00.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:57:00.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:57:00.665 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:57:00.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:57:00.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:57:00.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:57:00.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:57:00.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:57:00.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:57:00.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:57:00.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_not_primary td/ceph-helpers 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local dir=td/ceph-helpers 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: setup td/ceph-helpers 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:57:00.678 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:57:00.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:57:00.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:57:00.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:57:00.680 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:57:00.681 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:57:00.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:57:00.682 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:57:00.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:57:00.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:57:00.683 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:57:00.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:57:00.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:57:00.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:57:00.685 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:57:00.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:57:00.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:57:00.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:57:00.688 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:57:00.688 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:57:00.688 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:57:00.689 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:57:00.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:57:00.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:57:00.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:57:00.691 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:57:00.691 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:57:00.691 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:57:00.692 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:57:00.693 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:57:00.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:57:00.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:57:00.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:57:00.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-06-27T21:57:00.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:57:00.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:57:00.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:57:00.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:57:00.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:57:00.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-06-27T21:57:01.191 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:57:01.191 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:57:01.191 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:57:01.192 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:57:01.192 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:57:01.192 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:57:01.192 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:57:01.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-27T21:57:01.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:57:01.412 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:57:01.412 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:57:01.412 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:57:01.412 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:57:01.413 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:57:01.414 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:57:01.414 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:57:01.414 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:57:01.414 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:57:01.414 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:57:01.414 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:57:01.414 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:57:01.414 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:57:01.415 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:57:01.533 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:57:01.533 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:57:01.534 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:57:01.534 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:57:01.534 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:57:01.534 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:57:01.534 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:57:01.534 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:57:01.535 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:57:01.535 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:57:01.535 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:57:01.535 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:57:01.535 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:57:01.536 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:57:01.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: run_mgr td/ceph-helpers x 2024-06-27T21:57:01.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:57:01.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:57:01.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:57:01.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:57:01.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:57:01.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:57:02.571 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:57:02.571 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:57:02.571 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:57:02.572 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:57:02.572 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:57:02.572 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:57:02.572 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:57:02.573 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:57:02.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:57:02.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1243: test_get_not_primary: run_osd td/ceph-helpers 0 2024-06-27T21:57:02.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:57:02.644 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:57:02.645 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:57:02.645 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:57:02.645 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:57:02.645 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:57:02.645 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:57:02.645 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:57:02.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:57:02.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:57:02.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:57:02.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:57:02.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:57:02.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:57:02.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:57:02.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:57:02.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:57:02.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:57:02.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:57:02.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:57:02.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:57:02.647 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:57:02.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd3bde8d-9281-4522-9199-5610d6894ca6 2024-06-27T21:57:02.647 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 cd3bde8d-9281-4522-9199-5610d6894ca6 2024-06-27T21:57:02.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cd3bde8d-9281-4522-9199-5610d6894ca6' 2024-06-27T21:57:02.648 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:57:02.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAu4H1m8AGMJxAAgV6sPPWQmjGiuhARIlOOUA== 2024-06-27T21:57:02.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAu4H1m8AGMJxAAgV6sPPWQmjGiuhARIlOOUA=="}' 2024-06-27T21:57:02.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd3bde8d-9281-4522-9199-5610d6894ca6 -i td/ceph-helpers/0/new.json 2024-06-27T21:57:03.015 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:57:03.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T21:57:03.034 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAu4H1m8AGMJxAAgV6sPPWQmjGiuhARIlOOUA== --osd-uuid cd3bde8d-9281-4522-9199-5610d6894ca6 2024-06-27T21:57:03.067 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:03.061+0000 7f356def97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:03.092 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:03.089+0000 7f356def97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:03.093 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:03.089+0000 7f356def97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:03.094 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:03.089+0000 7f356def97c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:57:03.094 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:03.089+0000 7f356def97c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:57:03.094 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:03.089+0000 7f356def97c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:57:03.123 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:03.117+0000 7f356def97c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T21:57:09.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T21:57:09.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:57:09.706 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:57:09.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:57:09.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:57:10.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:57:10.310 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:57:10.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:57:10.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:57:10.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:57:10.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:57:10.343 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:10.337+0000 7fa9c76cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:10.366 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:10.361+0000 7fa9c76cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:10.368 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:10.365+0000 7fa9c76cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:10.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:57:10.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:57:10.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:57:10.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:57:10.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:57:10.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:10.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:57:10.751 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:57:10.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:10.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:10.957 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:10.953+0000 7fa9c76cf7c0 -1 Falling back to public interface 2024-06-27T21:57:11.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:12.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:12.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:12.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:57:12.224 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:57:12.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:12.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:12.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:13.546 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:13.541+0000 7fa9c76cf7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:57:13.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:13.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:13.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:57:13.712 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:57:13.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:13.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:14.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:15.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:15.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:15.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:57:15.271 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:57:15.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:15.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:15.765 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:16.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:16.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:16.767 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:57:16.767 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:57:16.767 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:16.767 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:17.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:18.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:18.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:18.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:57:18.213 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:57:18.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:18.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:18.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:19.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:19.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:19.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:57:19.763 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:57:19.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:19.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:20.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:21.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:21.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:21.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:57:21.201 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:57:21.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:21.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:21.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:22.733 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:22.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:22.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:57:22.734 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:57:22.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:22.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:23.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:24.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:24.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:24.205 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:57:24.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:57:24.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:24.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:24.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:25.733 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:25.733 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:25.733 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:57:25.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:57:25.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:25.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:26.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:27.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:27.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:27.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:57:27.191 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:57:27.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:27.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:27.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:28.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:28.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:28.646 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:57:28.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:57:28.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:28.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:29.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:30.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:30.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:30.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:57:30.368 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:57:30.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:30.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:31.197 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:32.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:32.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:32.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:57:32.199 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:57:32.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:32.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:33.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:34.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:34.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:34.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:57:34.272 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:57:34.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:34.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:35.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:36.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:36.425 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:36.425 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:57:36.425 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:57:36.425 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:36.426 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:37.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:38.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:38.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:38.311 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:57:38.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:57:38.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:38.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:57:38.804 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1713981735,v1:127.0.0.1:6803/1713981735] [v2:127.0.0.1:6804/1713981735,v1:127.0.0.1:6805/1713981735] exists,up cd3bde8d-9281-4522-9199-5610d6894ca6 2024-06-27T21:57:38.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:57:38.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:57:38.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:57:38.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: test_get_not_primary: run_osd td/ceph-helpers 1 2024-06-27T21:57:38.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T21:57:38.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:57:38.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T21:57:38.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:57:38.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-27T21:57:38.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:57:38.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:57:38.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:57:38.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:57:38.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-27T21:57:38.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-27T21:57:38.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:57:38.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:57:38.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:57:38.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:57:38.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:57:38.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:57:38.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:57:38.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:57:38.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:57:38.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:57:38.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-27T21:57:38.812 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:57:38.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=839b01d8-1984-4f89-9e7d-d66492d35629 2024-06-27T21:57:38.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 839b01d8-1984-4f89-9e7d-d66492d35629' 2024-06-27T21:57:38.813 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 839b01d8-1984-4f89-9e7d-d66492d35629 2024-06-27T21:57:38.814 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:57:38.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBS4H1mAgHGMRAAidXzNeIR25s/MJ5/kidS3Q== 2024-06-27T21:57:38.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBS4H1mAgHGMRAAidXzNeIR25s/MJ5/kidS3Q=="}' 2024-06-27T21:57:38.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 839b01d8-1984-4f89-9e7d-d66492d35629 -i td/ceph-helpers/1/new.json 2024-06-27T21:57:39.334 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:57:39.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-27T21:57:39.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBS4H1mAgHGMRAAidXzNeIR25s/MJ5/kidS3Q== --osd-uuid 839b01d8-1984-4f89-9e7d-d66492d35629 2024-06-27T21:57:39.378 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:39.373+0000 7f2a327f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:39.394 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:39.389+0000 7f2a327f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:39.395 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:39.389+0000 7f2a327f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:39.395 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:39.393+0000 7f2a327f37c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T21:57:39.395 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:39.393+0000 7f2a327f37c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T21:57:39.395 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:39.393+0000 7f2a327f37c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T21:57:39.433 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:39.429+0000 7f2a327f37c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-27T21:57:45.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-27T21:57:45.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:57:45.858 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T21:57:45.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T21:57:45.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:57:46.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T21:57:46.445 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T21:57:46.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:57:46.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:57:46.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:57:46.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:57:46.470 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:46.465+0000 7f50ae7ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:46.506 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:46.501+0000 7f50ae7ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:46.508 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:46.505+0000 7f50ae7ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:57:46.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T21:57:46.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:57:46.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T21:57:46.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:57:46.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:57:46.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:46.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:57:46.894 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:57:46.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:46.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:57:47.082 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:47.077+0000 7f50ae7ce7c0 -1 Falling back to public interface 2024-06-27T21:57:47.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:48.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:48.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:48.381 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:57:48.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:57:48.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:48.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:57:48.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:49.931 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:57:49.925+0000 7f50ae7ce7c0 -1 osd.1 0 log_to_monitors true 2024-06-27T21:57:49.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:49.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:49.958 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:57:49.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:57:49.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:49.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:57:50.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:51.696 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:51.696 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:51.696 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:57:51.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:57:51.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:51.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:57:52.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:53.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:53.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:53.335 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:57:53.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:57:53.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:53.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:57:54.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:55.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:55.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:55.031 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:57:55.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:57:55.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:55.032 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:57:55.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:56.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:56.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:56.465 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:57:56.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:57:56.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:56.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:57:56.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:57.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:57.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:57.946 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:57:57.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:57:57.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:57.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:57:58.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:57:59.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:57:59.396 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:57:59.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:57:59.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:57:59.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:57:59.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:57:59.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:00.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:00.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:00.999 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:58:00.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:58:00.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:00.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:58:01.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:02.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:02.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:02.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:58:02.470 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:58:02.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:02.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:58:02.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:03.930 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:03.930 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:58:03.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:03.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:58:03.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:03.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:58:05.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:06.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:06.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:06.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:58:06.036 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:58:06.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:06.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:58:07.345 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:08.347 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:08.347 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:08.347 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:58:08.348 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:58:08.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:08.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:58:09.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:10.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:10.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:10.216 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:58:10.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:58:10.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:10.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:58:11.124 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:12.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:12.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:12.126 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:58:12.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:58:12.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:12.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:58:13.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:14.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:14.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:14.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:58:14.283 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:58:14.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:14.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T21:58:14.793 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1300348531,v1:127.0.0.1:6811/1300348531] [v2:127.0.0.1:6812/1300348531,v1:127.0.0.1:6813/1300348531] exists,up 839b01d8-1984-4f89-9e7d-d66492d35629 2024-06-27T21:58:14.793 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:58:14.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:58:14.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:58:14.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: test_get_not_primary: create_rbd_pool 2024-06-27T21:58:14.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T21:58:15.287 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T21:58:15.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T21:58:15.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T21:58:15.876 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T21:58:15.892 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T21:58:16.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T21:58:19.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1246: test_get_not_primary: wait_for_clean 2024-06-27T21:58:19.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T21:58:19.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T21:58:19.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T21:58:19.055 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T21:58:19.056 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T21:58:19.056 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T21:58:19.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T21:58:19.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T21:58:19.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T21:58:19.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T21:58:19.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T21:58:19.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T21:58:19.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T21:58:19.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:58:19.183 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:58:19.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T21:58:19.661 INFO:tasks.workunit.client.0.smithi042.stderr:1' 2024-06-27T21:58:19.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:58:19.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:58:19.662 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:58:19.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836493 2024-06-27T21:58:19.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836493 2024-06-27T21:58:19.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493' 2024-06-27T21:58:19.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:58:19.959 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T21:58:20.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672966 2024-06-27T21:58:20.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672966 2024-06-27T21:58:20.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493 1-42949672966' 2024-06-27T21:58:20.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:58:20.127 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836493 2024-06-27T21:58:20.127 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:58:20.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:58:20.128 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836493 2024-06-27T21:58:20.128 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:58:20.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836493 2024-06-27T21:58:20.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-06-27T21:58:20.129 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836493 2024-06-27T21:58:20.129 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:58:20.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836491 -lt 21474836493 2024-06-27T21:58:20.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T21:58:21.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T21:58:21.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:58:22.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836493 -lt 21474836493 2024-06-27T21:58:22.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:58:22.100 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672966 2024-06-27T21:58:22.101 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:58:22.101 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T21:58:22.102 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672966 2024-06-27T21:58:22.103 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:58:22.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672966 2024-06-27T21:58:22.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-06-27T21:58:22.104 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672966 2024-06-27T21:58:22.105 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T21:58:22.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672967 -lt 42949672966 2024-06-27T21:58:22.551 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T21:58:22.552 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:58:22.552 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:58:23.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T21:58:23.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T21:58:23.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T21:58:23.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T21:58:23.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T21:58:23.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T21:58:23.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T21:58:23.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T21:58:23.776 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T21:58:23.777 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T21:58:23.777 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:58:23.777 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:58:24.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T21:58:24.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T21:58:24.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T21:58:24.328 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: get_primary rbd GROUP 2024-06-27T21:58:24.328 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-27T21:58:24.328 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-06-27T21:58:24.329 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-06-27T21:58:24.329 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-27T21:58:24.839 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: local primary=1 2024-06-27T21:58:24.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: get_not_primary rbd GROUP 2024-06-27T21:58:24.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-06-27T21:58:24.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=GROUP 2024-06-27T21:58:24.841 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd GROUP 2024-06-27T21:58:24.841 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-27T21:58:24.841 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-06-27T21:58:24.841 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-06-27T21:58:24.841 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-27T21:58:25.300 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-27T21:58:25.300 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-27T21:58:25.300 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd GROUP 2024-06-27T21:58:25.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: local not_primary=0 2024-06-27T21:58:25.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: test_get_not_primary: test 0 '!=' 1 2024-06-27T21:58:25.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: test_get_not_primary: test 0 = 0 -o 0 = 1 2024-06-27T21:58:25.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: test_get_not_primary: teardown td/ceph-helpers 2024-06-27T21:58:25.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:58:25.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:58:25.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:58:25.756 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:58:25.756 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:58:25.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:58:25.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:58:25.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:58:26.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:58:26.067 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:58:26.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:58:26.069 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:58:26.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:58:26.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:58:26.071 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:58:26.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:58:26.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:58:26.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:58:26.074 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:58:26.075 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:58:26.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:58:26.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:58:26.090 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:58:26.090 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:58:26.091 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:58:26.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_num_active_clean td/ceph-helpers 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: local dir=td/ceph-helpers 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1374: test_get_num_active_clean: setup td/ceph-helpers 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:58:26.093 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:58:26.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:58:26.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:58:26.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:58:26.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:58:26.096 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:58:26.097 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:58:26.098 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:58:26.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:58:26.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:58:26.099 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:58:26.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:58:26.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:58:26.101 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:58:26.101 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:58:26.102 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:58:26.103 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:58:26.103 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:58:26.105 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:58:26.105 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:58:26.105 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:58:26.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:58:26.106 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:58:26.106 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:58:26.106 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:58:26.108 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:58:26.108 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:58:26.108 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:58:26.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:58:26.110 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:58:26.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:58:26.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:58:26.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:58:26.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1375: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:58:26.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:58:26.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:58:26.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:58:26.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:58:26.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:58:26.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:58:26.622 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:58:26.622 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:58:26.622 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:58:26.623 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:58:26.623 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:58:26.623 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:58:26.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:58:26.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:58:26.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:58:26.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:58:26.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:58:26.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:58:26.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:58:26.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:58:26.844 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:58:26.844 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:58:26.844 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:58:26.845 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:58:26.845 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:58:26.845 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:58:26.845 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:58:26.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:58:26.846 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:58:26.963 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:58:26.963 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:58:26.963 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:58:26.963 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:58:26.964 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:58:26.964 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:58:26.964 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:58:26.964 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:58:26.965 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:58:26.965 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:58:26.965 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:58:26.965 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:58:26.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:58:26.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:58:27.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1376: test_get_num_active_clean: run_mgr td/ceph-helpers x 2024-06-27T21:58:27.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:58:27.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:58:27.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:58:27.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:58:27.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:58:27.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:58:28.009 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:58:28.009 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:58:28.009 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:58:28.010 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:58:28.010 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:58:28.010 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:58:28.010 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:58:28.010 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:58:28.011 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:58:28.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1377: test_get_num_active_clean: run_osd td/ceph-helpers 0 2024-06-27T21:58:28.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T21:58:28.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:58:28.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:58:28.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:58:28.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:58:28.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:58:28.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:58:28.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:58:28.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:58:28.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:58:28.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:58:28.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:58:28.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:58:28.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:58:28.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:58:28.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:58:28.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:58:28.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:58:28.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:58:28.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:58:28.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:58:28.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:58:28.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:58:28.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:58:28.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d073f72-74f4-4062-ba0f-25227f0f8c72 2024-06-27T21:58:28.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2d073f72-74f4-4062-ba0f-25227f0f8c72' 2024-06-27T21:58:28.088 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 2d073f72-74f4-4062-ba0f-25227f0f8c72 2024-06-27T21:58:28.088 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:58:28.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCE4H1mQmw/BhAAyp1B9UE5YlyHeJihKzZ+WQ== 2024-06-27T21:58:28.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCE4H1mQmw/BhAAyp1B9UE5YlyHeJihKzZ+WQ=="}' 2024-06-27T21:58:28.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d073f72-74f4-4062-ba0f-25227f0f8c72 -i td/ceph-helpers/0/new.json 2024-06-27T21:58:28.421 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:58:28.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T21:58:28.438 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCE4H1mQmw/BhAAyp1B9UE5YlyHeJihKzZ+WQ== --osd-uuid 2d073f72-74f4-4062-ba0f-25227f0f8c72 2024-06-27T21:58:28.473 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:28.468+0000 7f3aca9327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:58:28.489 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:28.484+0000 7f3aca9327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:58:28.490 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:28.484+0000 7f3aca9327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:58:28.491 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:28.484+0000 7f3aca9327c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:58:28.491 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:28.484+0000 7f3aca9327c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:58:28.491 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:28.484+0000 7f3aca9327c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:58:28.529 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:28.524+0000 7f3aca9327c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T21:58:35.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T21:58:35.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:58:35.215 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:58:35.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:58:35.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:58:35.817 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:58:35.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:58:35.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:58:35.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:58:35.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:58:35.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:58:35.850 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:35.844+0000 7fa07528e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:58:35.862 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:35.856+0000 7fa07528e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:58:35.863 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:35.856+0000 7fa07528e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:58:36.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:58:36.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:58:36.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:58:36.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:58:36.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:58:36.235 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:58:36.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:36.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:58:36.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:36.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:36.437 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:36.432+0000 7fa07528e7c0 -1 Falling back to public interface 2024-06-27T21:58:36.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:37.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:37.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:37.686 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:58:37.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:58:37.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:37.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:38.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:39.100 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:58:39.096+0000 7fa07528e7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:58:39.130 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:58:39.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:39.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:39.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:58:39.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:39.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:39.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:40.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:40.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:40.806 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:58:40.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:58:40.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:40.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:41.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:42.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:42.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:42.289 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:58:42.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:58:42.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:42.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:42.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:43.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:43.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:43.743 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:58:43.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:58:43.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:43.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:44.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:45.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:45.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:45.344 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:58:45.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:58:45.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:45.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:45.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:46.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:46.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:46.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:58:46.972 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:58:46.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:46.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:47.427 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:48.429 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:58:48.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:58:48.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:48.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:48.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:49.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:49.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:49.935 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:58:49.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:58:49.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:49.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:50.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:51.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:51.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:51.393 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:58:51.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:58:51.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:51.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:52.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:53.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:53.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:53.182 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:58:53.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:58:53.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:53.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:54.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:55.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:55.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:55.273 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:58:55.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:58:55.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:55.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:56.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:57.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:57.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:57.023 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:58:57.023 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:58:57.023 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:57.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:57.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:58:58.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:58:58.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:58:58.896 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:58:58.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:58:58.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:58:58.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:58:59.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:00.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:00.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:00.592 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:59:00.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:59:00.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:00.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:01.257 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:02.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:02.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:02.259 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:59:02.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:59:02.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:02.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:02.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:03.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:03.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:03.843 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:59:03.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:59:03.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:03.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:04.312 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/595951653,v1:127.0.0.1:6803/595951653] [v2:127.0.0.1:6804/595951653,v1:127.0.0.1:6805/595951653] exists,up 2d073f72-74f4-4062-ba0f-25227f0f8c72 2024-06-27T21:59:04.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:59:04.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:59:04.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:59:04.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1378: test_get_num_active_clean: create_rbd_pool 2024-06-27T21:59:04.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T21:59:04.787 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T21:59:04.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T21:59:04.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T21:59:05.443 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T21:59:05.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T21:59:06.462 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T21:59:07.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1379: test_get_num_active_clean: wait_for_clean 2024-06-27T21:59:07.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T21:59:07.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T21:59:07.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T21:59:07.611 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T21:59:07.612 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T21:59:07.612 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T21:59:07.613 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T21:59:07.613 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T21:59:07.613 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T21:59:07.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T21:59:07.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T21:59:07.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T21:59:07.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T21:59:07.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:59:07.739 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:59:08.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T21:59:08.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:59:08.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:59:08.221 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:59:08.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836489 2024-06-27T21:59:08.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836489 2024-06-27T21:59:08.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489' 2024-06-27T21:59:08.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:59:08.405 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:59:08.405 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836489 2024-06-27T21:59:08.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:59:08.407 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836489 2024-06-27T21:59:08.407 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:59:08.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836489 2024-06-27T21:59:08.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-06-27T21:59:08.408 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836489 2024-06-27T21:59:08.408 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:59:08.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836489 2024-06-27T21:59:08.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T21:59:09.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T21:59:09.890 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:59:10.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836490 -lt 21474836489 2024-06-27T21:59:10.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T21:59:10.364 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:59:10.364 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:59:10.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T21:59:10.941 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T21:59:10.941 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T21:59:10.942 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T21:59:10.942 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T21:59:10.942 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T21:59:10.942 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T21:59:10.942 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T21:59:11.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T21:59:11.411 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T21:59:11.412 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:59:11.412 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:59:11.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T21:59:11.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T21:59:11.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T21:59:11.977 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: get_num_active_clean 2024-06-27T21:59:11.977 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T21:59:11.977 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T21:59:11.977 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T21:59:11.977 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T21:59:11.977 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T21:59:12.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: local num_active_clean=4 2024-06-27T21:59:12.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1381: test_get_num_active_clean: test 4 = 4 2024-06-27T21:59:12.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1382: test_get_num_active_clean: teardown td/ceph-helpers 2024-06-27T21:59:12.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:59:12.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:59:12.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:59:12.415 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:59:12.416 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:59:12.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:59:12.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:59:12.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:59:12.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:59:12.524 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:59:12.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:59:12.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:59:12.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:59:12.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:59:12.526 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:59:12.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:59:12.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:59:12.527 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:59:12.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:59:12.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:59:12.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:59:12.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:59:12.535 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:59:12.535 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:12.535 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:12.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:59:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:59:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:59:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:59:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_num_active_or_peered td/ceph-helpers 2024-06-27T21:59:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: local dir=td/ceph-helpers 2024-06-27T21:59:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1404: test_get_num_active_or_peered: setup td/ceph-helpers 2024-06-27T21:59:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:59:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:59:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:59:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:59:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:59:12.538 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:59:12.538 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:59:12.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:59:12.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:59:12.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:59:12.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:59:12.538 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:59:12.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:59:12.539 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:59:12.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:59:12.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:59:12.540 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:59:12.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:59:12.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:59:12.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:59:12.541 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:59:12.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:59:12.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:59:12.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:59:12.544 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:59:12.544 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:12.544 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:12.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:59:12.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:59:12.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:59:12.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:59:12.546 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:59:12.546 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:12.546 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:12.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:59:12.548 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:59:12.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:59:12.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:59:12.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:59:12.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1405: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:59:12.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:59:12.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:59:12.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:59:12.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:59:12.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:59:12.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:59:13.055 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:59:13.056 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:59:13.056 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:59:13.056 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:59:13.056 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:13.057 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:13.057 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:59:13.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:59:13.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:59:13.278 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:59:13.278 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:59:13.278 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:59:13.278 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:59:13.282 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:59:13.282 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:59:13.283 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:59:13.283 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:59:13.283 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:59:13.283 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:13.283 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:13.283 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:59:13.283 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:59:13.283 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:59:13.399 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:59:13.399 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:59:13.399 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:59:13.399 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:59:13.400 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:59:13.400 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:59:13.400 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:59:13.400 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:59:13.401 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:59:13.401 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:13.401 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:13.402 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:59:13.402 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:59:13.402 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:59:13.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1406: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2024-06-27T21:59:13.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:59:13.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:59:13.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:59:13.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:59:13.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:59:13.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T21:59:14.432 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T21:59:14.432 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:59:14.432 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:59:14.433 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:59:14.433 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:14.433 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:14.433 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:59:14.434 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:59:14.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T21:59:14.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1407: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T21:59:14.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T21:59:14.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T21:59:14.510 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T21:59:14.510 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:59:14.510 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:59:14.510 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:59:14.510 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:14.510 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:14.510 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:59:14.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:59:14.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T21:59:14.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T21:59:14.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ac483ed4-2efa-4f7f-9d9a-3fdf88dbc275 2024-06-27T21:59:14.513 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 ac483ed4-2efa-4f7f-9d9a-3fdf88dbc275 2024-06-27T21:59:14.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ac483ed4-2efa-4f7f-9d9a-3fdf88dbc275' 2024-06-27T21:59:14.513 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T21:59:14.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCy4H1mZsCpHxAAn0IRoowPGzCVGr2pxf4Vgw== 2024-06-27T21:59:14.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCy4H1mZsCpHxAAn0IRoowPGzCVGr2pxf4Vgw=="}' 2024-06-27T21:59:14.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ac483ed4-2efa-4f7f-9d9a-3fdf88dbc275 -i td/ceph-helpers/0/new.json 2024-06-27T21:59:14.855 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:59:14.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T21:59:14.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCy4H1mZsCpHxAAn0IRoowPGzCVGr2pxf4Vgw== --osd-uuid ac483ed4-2efa-4f7f-9d9a-3fdf88dbc275 2024-06-27T21:59:14.915 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:14.912+0000 7fb7dec587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:59:14.932 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:14.932+0000 7fb7dec587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:59:14.933 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:14.932+0000 7fb7dec587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:59:14.934 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:14.932+0000 7fb7dec587c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:59:14.934 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:14.932+0000 7fb7dec587c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:59:14.934 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:14.932+0000 7fb7dec587c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T21:59:14.963 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:14.964+0000 7fb7dec587c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T21:59:21.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T21:59:21.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T21:59:21.603 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T21:59:21.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T21:59:21.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T21:59:22.193 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T21:59:22.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T21:59:22.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T21:59:22.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T21:59:22.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T21:59:22.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T21:59:22.226 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:22.224+0000 7f6ee660b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:59:22.229 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:22.228+0000 7f6ee660b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:59:22.231 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:22.228+0000 7f6ee660b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T21:59:22.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T21:59:22.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T21:59:22.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T21:59:22.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T21:59:22.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T21:59:22.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:22.669 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T21:59:22.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T21:59:22.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:22.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:22.782 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:22.780+0000 7f6ee660b7c0 -1 Falling back to public interface 2024-06-27T21:59:23.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:24.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:24.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:24.115 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T21:59:24.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T21:59:24.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:24.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:24.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:25.437 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T21:59:25.436+0000 7f6ee660b7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T21:59:25.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:25.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:25.563 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T21:59:25.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T21:59:25.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:25.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:26.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:27.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:27.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:27.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T21:59:27.115 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T21:59:27.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:27.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:27.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:28.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:28.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:28.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T21:59:28.554 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T21:59:28.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:28.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:28.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:29.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:29.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:29.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T21:59:29.997 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T21:59:29.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:29.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:30.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:31.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:31.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:31.584 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T21:59:31.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T21:59:31.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:31.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:32.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:33.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:33.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:33.039 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T21:59:33.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T21:59:33.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:33.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:33.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:34.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:34.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:34.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T21:59:34.596 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T21:59:34.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:34.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:35.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:36.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:36.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:36.043 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T21:59:36.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T21:59:36.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:36.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:36.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:37.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:37.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:37.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T21:59:37.575 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T21:59:37.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:37.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:38.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:39.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:39.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:39.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T21:59:39.022 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T21:59:39.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:39.023 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:39.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:40.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:40.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:40.942 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T21:59:40.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T21:59:40.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:40.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:41.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:42.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:42.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:42.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T21:59:42.789 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T21:59:42.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:42.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:43.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:44.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:44.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:44.636 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T21:59:44.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T21:59:44.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:44.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:45.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:46.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:46.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:46.468 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T21:59:46.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T21:59:46.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:46.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:47.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:48.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:48.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:48.184 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T21:59:48.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T21:59:48.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:48.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:49.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T21:59:50.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T21:59:50.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T21:59:50.143 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T21:59:50.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T21:59:50.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T21:59:50.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T21:59:50.584 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3611785062,v1:127.0.0.1:6803/3611785062] [v2:127.0.0.1:6804/3611785062,v1:127.0.0.1:6805/3611785062] exists,up ac483ed4-2efa-4f7f-9d9a-3fdf88dbc275 2024-06-27T21:59:50.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T21:59:50.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T21:59:50.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T21:59:50.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1408: test_get_num_active_or_peered: create_rbd_pool 2024-06-27T21:59:50.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T21:59:51.083 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T21:59:51.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T21:59:51.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T21:59:51.644 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T21:59:51.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T21:59:52.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T21:59:53.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1409: test_get_num_active_or_peered: wait_for_clean 2024-06-27T21:59:53.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T21:59:53.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T21:59:53.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T21:59:53.861 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T21:59:53.862 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T21:59:53.862 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T21:59:53.863 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T21:59:53.863 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T21:59:53.863 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T21:59:53.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T21:59:53.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T21:59:53.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T21:59:53.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T21:59:53.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T21:59:53.989 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T21:59:54.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T21:59:54.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T21:59:54.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T21:59:54.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T21:59:54.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836486 2024-06-27T21:59:54.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836486 2024-06-27T21:59:54.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836486' 2024-06-27T21:59:54.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T21:59:54.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836486 2024-06-27T21:59:54.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T21:59:54.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T21:59:54.608 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836486 2024-06-27T21:59:54.608 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T21:59:54.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836486 2024-06-27T21:59:54.609 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836486 2024-06-27T21:59:54.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-06-27T21:59:54.610 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:59:55.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836484 -lt 21474836486 2024-06-27T21:59:55.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T21:59:56.034 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T21:59:56.035 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T21:59:56.476 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836486 2024-06-27T21:59:56.477 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T21:59:56.477 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:59:56.477 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:59:57.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T21:59:57.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T21:59:57.051 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T21:59:57.051 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T21:59:57.051 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T21:59:57.051 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T21:59:57.052 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T21:59:57.052 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T21:59:57.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T21:59:57.472 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T21:59:57.472 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T21:59:57.472 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T21:59:58.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T21:59:58.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T21:59:58.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T21:59:58.048 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: get_num_active_or_peered 2024-06-27T21:59:58.048 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-06-27T21:59:58.049 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-06-27T21:59:58.049 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-06-27T21:59:58.049 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-06-27T21:59:58.049 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-06-27T21:59:58.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: local num_peered=4 2024-06-27T21:59:58.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1411: test_get_num_active_or_peered: test 4 = 4 2024-06-27T21:59:58.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1412: test_get_num_active_or_peered: teardown td/ceph-helpers 2024-06-27T21:59:58.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:59:58.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:59:58.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:59:58.486 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:59:58.486 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:59:58.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:59:58.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:59:58.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:59:58.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:59:58.594 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:59:58.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:59:58.595 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:59:58.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:59:58.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:59:58.596 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:59:58.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:59:58.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:59:58.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:59:58.598 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:59:58.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:59:58.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:59:58.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:59:58.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:59:58.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:58.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:58.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_num_pgs td/ceph-helpers 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: local dir=td/ceph-helpers 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1431: test_get_num_pgs: setup td/ceph-helpers 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T21:59:58.608 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T21:59:58.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T21:59:58.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T21:59:58.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T21:59:58.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T21:59:58.609 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T21:59:58.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T21:59:58.610 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T21:59:58.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T21:59:58.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T21:59:58.612 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T21:59:58.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:59:58.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T21:59:58.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T21:59:58.613 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T21:59:58.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T21:59:58.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T21:59:58.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T21:59:58.616 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T21:59:58.616 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:58.616 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:58.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T21:59:58.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T21:59:58.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T21:59:58.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T21:59:58.618 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T21:59:58.618 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:58.618 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:58.619 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T21:59:58.620 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T21:59:58.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T21:59:58.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T21:59:58.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T21:59:58.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:59:58.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T21:59:58.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T21:59:58.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T21:59:58.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T21:59:58.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T21:59:58.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:59:59.105 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T21:59:59.105 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T21:59:59.105 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T21:59:59.106 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T21:59:59.106 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:59.106 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:59.106 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T21:59:59.106 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T21:59:59.309 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T21:59:59.310 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T21:59:59.310 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:59:59.310 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:59:59.310 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T21:59:59.311 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T21:59:59.311 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:59:59.311 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:59:59.311 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:59:59.312 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:59:59.312 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:59.312 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:59.312 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:59:59.312 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:59:59.312 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T21:59:59.430 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T21:59:59.431 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T21:59:59.431 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T21:59:59.431 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T21:59:59.431 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T21:59:59.431 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T21:59:59.432 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T21:59:59.432 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T21:59:59.432 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T21:59:59.432 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T21:59:59.432 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T21:59:59.433 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T21:59:59.433 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T21:59:59.433 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T21:59:59.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: test_get_num_pgs: run_mgr td/ceph-helpers x 2024-06-27T21:59:59.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T21:59:59.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T21:59:59.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T21:59:59.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T21:59:59.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T21:59:59.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:00:00.467 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:00:00.468 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:00:00.468 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:00:00.468 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:00:00.468 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:00.468 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:00.468 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:00:00.469 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:00:00.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:00:00.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: test_get_num_pgs: run_osd td/ceph-helpers 0 2024-06-27T22:00:00.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:00:00.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:00:00.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:00:00.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:00:00.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:00:00.542 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:00:00.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:00:00.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:00:00.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:00:00.544 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:00:00.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e5ff28a-e6d1-41f6-b85b-85b4be7ed1ef 2024-06-27T22:00:00.544 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 9e5ff28a-e6d1-41f6-b85b-85b4be7ed1ef 2024-06-27T22:00:00.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9e5ff28a-e6d1-41f6-b85b-85b4be7ed1ef' 2024-06-27T22:00:00.545 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:00:00.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDg4H1mE/mOIRAALtf9HRXD4KV6XxeiPO587w== 2024-06-27T22:00:00.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDg4H1mE/mOIRAALtf9HRXD4KV6XxeiPO587w=="}' 2024-06-27T22:00:00.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e5ff28a-e6d1-41f6-b85b-85b4be7ed1ef -i td/ceph-helpers/0/new.json 2024-06-27T22:00:00.920 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:00:00.937 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:00:00.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDg4H1mE/mOIRAALtf9HRXD4KV6XxeiPO587w== --osd-uuid 9e5ff28a-e6d1-41f6-b85b-85b4be7ed1ef 2024-06-27T22:00:00.971 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:00.967+0000 7eff1bf877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:00.997 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:00.995+0000 7eff1bf877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:00.998 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:00.995+0000 7eff1bf877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:00.998 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:00.995+0000 7eff1bf877c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:00:00.999 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:00.995+0000 7eff1bf877c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:00:00.999 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:00.995+0000 7eff1bf877c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:00:01.037 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:01.035+0000 7eff1bf877c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:00:07.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:00:07.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:00:07.680 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:00:07.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:00:07.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:00:08.266 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:00:08.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:00:08.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:00:08.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:00:08.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:00:08.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:00:08.298 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:08.295+0000 7f2e328a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:08.330 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:08.327+0000 7f2e328a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:08.332 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:08.331+0000 7f2e328a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:08.706 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:00:08.706 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:00:08.706 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:00:08.706 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:00:08.706 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:00:08.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:08.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:00:08.707 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:00:08.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:08.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:08.889 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:08.887+0000 7f2e328a17c0 -1 Falling back to public interface 2024-06-27T22:00:09.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:10.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:10.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:10.179 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:00:10.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:00:10.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:10.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:10.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:11.513 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:11.511+0000 7f2e328a17c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:00:11.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:11.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:11.627 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:00:11.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:00:11.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:11.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:12.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:13.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:13.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:13.268 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:00:13.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:00:13.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:13.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:13.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:14.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:14.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:14.711 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:00:14.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:00:14.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:14.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:15.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:16.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:16.161 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:00:16.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:16.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:00:16.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:16.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:16.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:17.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:17.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:17.743 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:00:17.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:00:17.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:17.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:18.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:19.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:19.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:19.190 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:00:19.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:00:19.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:19.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:19.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:20.753 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:20.753 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:20.754 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:00:20.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:00:20.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:20.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:21.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:22.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:22.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:22.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:00:22.208 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:00:22.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:22.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:22.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:23.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:23.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:23.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:00:23.808 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:00:23.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:23.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:24.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:25.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:25.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:25.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:00:25.425 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:00:25.425 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:25.425 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:26.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:27.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:27.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:27.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:00:27.357 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:00:27.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:27.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:28.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:29.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:29.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:29.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:00:29.420 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:00:29.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:29.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:30.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:31.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:31.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:31.165 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:00:31.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:00:31.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:31.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:31.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:32.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:32.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:32.978 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:00:32.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:00:32.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:32.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:34.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:35.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:35.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:35.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:00:35.057 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:00:35.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:35.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:35.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:36.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:36.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:36.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:00:36.529 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:00:36.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:36.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:36.965 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1558243569,v1:127.0.0.1:6803/1558243569] [v2:127.0.0.1:6804/1558243569,v1:127.0.0.1:6805/1558243569] exists,up 9e5ff28a-e6d1-41f6-b85b-85b4be7ed1ef 2024-06-27T22:00:36.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:00:36.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:00:36.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:00:36.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: test_get_num_pgs: create_rbd_pool 2024-06-27T22:00:36.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:00:37.391 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:00:37.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:00:37.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:00:38.013 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:00:38.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:00:39.032 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:00:40.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1436: test_get_num_pgs: wait_for_clean 2024-06-27T22:00:40.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:00:40.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:00:40.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:00:40.404 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:00:40.405 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:00:40.405 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:00:40.406 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:00:40.406 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:00:40.406 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:00:40.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:00:40.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:00:40.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:00:40.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:00:40.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:00:40.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:00:41.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:00:41.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:00:41.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:00:41.019 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:00:41.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836488 2024-06-27T22:00:41.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836488 2024-06-27T22:00:41.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836488' 2024-06-27T22:00:41.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:00:41.194 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836488 2024-06-27T22:00:41.194 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:00:41.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:00:41.195 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836488 2024-06-27T22:00:41.195 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:00:41.196 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836488 2024-06-27T22:00:41.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836488 2024-06-27T22:00:41.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-06-27T22:00:41.196 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:00:41.663 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836488 -lt 21474836488 2024-06-27T22:00:41.664 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:00:41.664 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:00:41.664 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:00:42.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:00:42.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:00:42.254 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:00:42.254 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:00:42.254 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:00:42.254 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:00:42.254 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:00:42.255 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:00:42.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:00:42.688 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:00:42.688 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:00:42.688 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:00:43.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:00:43.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:00:43.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:00:43.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: get_num_pgs 2024-06-27T22:00:43.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:00:43.249 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:00:43.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: local num_pgs=4 2024-06-27T22:00:43.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1438: test_get_num_pgs: test 4 -gt 0 2024-06-27T22:00:43.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1439: test_get_num_pgs: teardown td/ceph-helpers 2024-06-27T22:00:43.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:00:43.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:00:43.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:00:43.837 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:00:43.837 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:00:43.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:00:43.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:00:43.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:00:43.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:00:43.943 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:00:43.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:00:43.944 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:00:43.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:00:43.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:00:43.945 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:00:43.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:00:43.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:00:43.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:00:43.946 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:00:43.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:00:43.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:00:43.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:00:43.953 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:00:43.954 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:43.954 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:43.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:00:43.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_op_scheduler td/ceph-helpers 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: test_get_op_scheduler: local dir=td/ceph-helpers 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: test_get_op_scheduler: setup td/ceph-helpers 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:00:43.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:00:43.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:00:43.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:00:43.957 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:00:43.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:00:43.958 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:00:43.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:00:43.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:00:43.959 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:00:43.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:00:43.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:00:43.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:00:43.960 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:00:43.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:00:43.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:00:43.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:00:43.962 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:00:43.963 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:43.963 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:43.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:00:43.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:00:43.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:00:43.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:00:43.964 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:00:43.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:43.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:43.965 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:00:43.966 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:00:43.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:00:43.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:00:43.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:00:43.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: test_get_op_scheduler: run_mon td/ceph-helpers a 2024-06-27T22:00:43.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:00:43.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:00:43.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:00:43.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:00:43.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:00:43.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-27T22:00:44.428 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:00:44.429 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:00:44.429 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:00:44.429 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:00:44.429 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:44.429 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:44.430 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:00:44.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T22:00:44.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:00:44.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:00:44.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:00:44.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:00:44.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:00:44.642 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:00:44.642 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:00:44.642 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:00:44.642 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:00:44.643 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:00:44.643 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:44.643 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:44.643 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:00:44.643 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:00:44.643 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:00:44.762 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:00:44.762 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:00:44.762 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:00:44.762 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:00:44.763 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:00:44.764 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:00:44.764 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:00:44.764 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:00:44.764 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:00:44.764 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:44.764 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:44.764 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:00:44.764 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:00:44.764 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:00:44.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: test_get_op_scheduler: run_mgr td/ceph-helpers x 2024-06-27T22:00:44.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:00:44.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:00:44.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:00:44.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:00:44.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:00:44.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:00:45.783 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:00:45.783 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:00:45.783 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:00:45.783 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:00:45.783 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:45.783 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:45.783 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:00:45.784 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:00:45.785 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:00:45.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2024-06-27T22:00:45.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:00:45.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:00:45.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:00:45.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:00:45.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:00:45.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:00:45.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:00:45.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:00:45.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:00:45.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=wpq 2024-06-27T22:00:45.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:00:45.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:00:45.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=76aebff5-f00a-46fa-a15d-9cf1b2d59528 2024-06-27T22:00:45.852 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 76aebff5-f00a-46fa-a15d-9cf1b2d59528 2024-06-27T22:00:45.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 76aebff5-f00a-46fa-a15d-9cf1b2d59528' 2024-06-27T22:00:45.853 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:00:45.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAN4X1msJ7RMxAAhCyyLsoaAMQ69tg7jmmGqg== 2024-06-27T22:00:45.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAN4X1msJ7RMxAAhCyyLsoaAMQ69tg7jmmGqg=="}' 2024-06-27T22:00:45.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 76aebff5-f00a-46fa-a15d-9cf1b2d59528 -i td/ceph-helpers/0/new.json 2024-06-27T22:00:46.194 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:00:46.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:00:46.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQAN4X1msJ7RMxAAhCyyLsoaAMQ69tg7jmmGqg== --osd-uuid 76aebff5-f00a-46fa-a15d-9cf1b2d59528 2024-06-27T22:00:46.255 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:46.251+0000 7fd59ad837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:46.271 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:46.267+0000 7fd59ad837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:46.272 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:46.271+0000 7fd59ad837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:46.273 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:46.271+0000 7fd59ad837c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:00:46.273 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:46.271+0000 7fd59ad837c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:00:46.273 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:46.271+0000 7fd59ad837c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:00:46.311 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:46.307+0000 7fd59ad837c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:00:52.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:00:52.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:00:52.998 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:00:52.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:00:52.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:00:53.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:00:53.577 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:00:53.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2024-06-27T22:00:53.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:00:53.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:00:53.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:00:53.610 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:53.607+0000 7f0eb745f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:53.620 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:53.619+0000 7f0eb745f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:53.622 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:53.619+0000 7f0eb745f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:54.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:00:54.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:00:54.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:00:54.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:00:54.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:00:54.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:54.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:00:54.022 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:00:54.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:54.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:54.193 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:54.191+0000 7f0eb745f7c0 -1 Falling back to public interface 2024-06-27T22:00:54.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:55.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:55.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:55.456 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:00:55.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:00:55.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:55.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:55.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:56.879 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:56.875+0000 7f0eb745f7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:00:56.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:56.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:56.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:00:56.899 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:00:56.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:56.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:57.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:00:58.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:00:58.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:00:58.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:00:58.395 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:00:58.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:00:58.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:00:58.897 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1151317413,v1:127.0.0.1:6803/1151317413] [v2:127.0.0.1:6804/1151317413,v1:127.0.0.1:6805/1151317413] exists,up 76aebff5-f00a-46fa-a15d-9cf1b2d59528 2024-06-27T22:00:58.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:00:58.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:00:58.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:00:58.899 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: test_get_op_scheduler: get_op_scheduler 0 2024-06-27T22:00:58.899 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: get_op_scheduler: local id=0 2024-06-27T22:00:58.899 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: get_op_scheduler: get_config osd 0 osd_op_queue 2024-06-27T22:00:58.899 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-06-27T22:00:58.899 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-06-27T22:00:58.899 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-06-27T22:00:58.900 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-06-27T22:00:58.900 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-06-27T22:00:58.900 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T22:00:58.900 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T22:00:58.901 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:00:58.901 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:58.901 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:58.901 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.0.asok 2024-06-27T22:00:58.901 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:00:58.901 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-osd.0.asok config get osd_op_queue 2024-06-27T22:00:59.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: test_get_op_scheduler: test wpq = wpq 2024-06-27T22:00:59.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2024-06-27T22:00:59.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:00:59.027 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:00:59.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:00:59.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:00:59.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:00:59.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-06-27T22:00:59.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-27T22:00:59.029 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:00:59.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=38e9d060-6abe-4c9f-a65e-dbf9f72dba16 2024-06-27T22:00:59.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 38e9d060-6abe-4c9f-a65e-dbf9f72dba16' 2024-06-27T22:00:59.030 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 38e9d060-6abe-4c9f-a65e-dbf9f72dba16 2024-06-27T22:00:59.030 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:00:59.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAb4X1ma6gYAxAApmzI6rf+wNkod1e+ENNoxw== 2024-06-27T22:00:59.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAb4X1ma6gYAxAApmzI6rf+wNkod1e+ENNoxw=="}' 2024-06-27T22:00:59.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 38e9d060-6abe-4c9f-a65e-dbf9f72dba16 -i td/ceph-helpers/1/new.json 2024-06-27T22:00:59.559 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:00:59.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-27T22:00:59.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAb4X1ma6gYAxAApmzI6rf+wNkod1e+ENNoxw== --osd-uuid 38e9d060-6abe-4c9f-a65e-dbf9f72dba16 2024-06-27T22:00:59.612 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:59.611+0000 7f227cf187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:59.624 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:59.623+0000 7f227cf187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:59.625 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:59.623+0000 7f227cf187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:00:59.626 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:59.623+0000 7f227cf187c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:00:59.626 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:59.623+0000 7f227cf187c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:00:59.626 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:59.623+0000 7f227cf187c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:00:59.662 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:00:59.659+0000 7f227cf187c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-27T22:01:06.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-27T22:01:06.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:01:06.281 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:01:06.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:01:06.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:01:06.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:01:06.888 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:01:06.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-06-27T22:01:06.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:01:06.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:01:06.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:01:06.922 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:06.919+0000 7f6ffbec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:01:06.937 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:06.935+0000 7f6ffbec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:01:06.939 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:06.935+0000 7f6ffbec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:01:07.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:01:07.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:01:07.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:01:07.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:01:07.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:01:07.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:07.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:01:07.323 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:01:07.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:07.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:07.498 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:07.495+0000 7f6ffbec67c0 -1 Falling back to public interface 2024-06-27T22:01:07.776 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:08.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:08.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:08.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:01:08.779 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:01:08.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:08.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:09.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:10.163 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:10.159+0000 7f6ffbec67c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:01:10.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:10.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:10.253 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:01:10.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:01:10.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:10.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:10.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:11.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:11.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:11.969 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:01:11.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:01:11.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:11.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:12.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:13.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:13.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:13.445 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:01:13.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:01:13.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:13.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:13.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:14.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:14.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:14.900 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:01:14.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:01:14.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:14.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:15.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:16.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:16.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:16.525 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:01:16.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:01:16.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:16.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:17.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:18.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:18.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:18.172 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:01:18.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:01:18.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:18.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:18.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:19.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:19.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:19.659 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:01:19.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:01:19.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:19.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:20.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:21.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:21.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:21.254 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:01:21.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:01:21.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:21.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:21.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:22.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:22.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:22.703 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:01:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:01:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:23.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:24.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:24.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:24.312 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:01:24.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:01:24.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:24.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:24.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:25.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:25.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:25.829 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:01:25.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:01:25.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:25.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:26.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:27.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:27.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:27.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:01:27.638 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:01:27.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:27.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:28.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:29.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:29.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:29.650 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:01:29.651 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:01:29.651 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:29.651 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:30.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:31.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:31.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:31.495 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:01:31.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:01:31.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:31.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:32.134 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:33.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:33.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:33.136 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:01:33.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:01:33.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:33.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:34.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:35.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:35.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:35.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:01:35.181 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:01:35.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:35.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:01:35.683 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3219062488,v1:127.0.0.1:6811/3219062488] [v2:127.0.0.1:6812/3219062488,v1:127.0.0.1:6813/3219062488] exists,up 38e9d060-6abe-4c9f-a65e-dbf9f72dba16 2024-06-27T22:01:35.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:01:35.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:01:35.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:01:35.685 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: test_get_op_scheduler: get_op_scheduler 1 2024-06-27T22:01:35.685 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: get_op_scheduler: local id=1 2024-06-27T22:01:35.685 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: get_op_scheduler: get_config osd 1 osd_op_queue 2024-06-27T22:01:35.685 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-06-27T22:01:35.685 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=1 2024-06-27T22:01:35.685 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-06-27T22:01:35.686 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-06-27T22:01:35.687 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.1 2024-06-27T22:01:35.687 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-27T22:01:35.687 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-27T22:01:35.687 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:01:35.687 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:01:35.687 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:01:35.687 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.1.asok 2024-06-27T22:01:35.687 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:01:35.687 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-osd.1.asok config get osd_op_queue 2024-06-27T22:01:35.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2024-06-27T22:01:35.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: test_get_op_scheduler: teardown td/ceph-helpers 2024-06-27T22:01:35.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:01:35.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:01:35.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:01:35.806 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:01:35.806 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:01:35.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:01:35.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:01:35.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:01:36.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:01:36.117 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:01:36.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:01:36.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:01:36.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:01:36.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:01:36.121 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:01:36.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:01:36.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:01:36.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:01:36.123 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:01:36.124 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:01:36.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:01:36.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:01:36.137 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:01:36.137 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:01:36.137 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:01:36.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:01:36.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:01:36.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:01:36.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:01:36.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2024-06-27T22:01:36.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-06-27T22:01:36.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1461: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2024-06-27T22:01:36.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:01:36.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:01:36.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:01:36.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:01:36.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:01:36.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:01:36.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:01:36.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:01:36.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:01:36.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:01:36.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:01:36.142 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:01:36.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:01:36.143 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:01:36.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:01:36.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:01:36.145 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:01:36.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:01:36.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:01:36.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:01:36.147 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:01:36.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:01:36.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:01:36.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:01:36.150 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:01:36.151 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:01:36.151 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:01:36.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:01:36.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:01:36.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:01:36.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:01:36.154 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:01:36.154 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:01:36.154 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:01:36.154 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:01:36.156 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:01:36.156 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:01:36.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:01:36.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:01:36.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1462: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:01:36.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:01:36.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:01:36.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:01:36.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:01:36.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:01:36.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:01:36.646 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:01:36.646 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:01:36.646 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:01:36.647 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:01:36.647 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:01:36.647 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:01:36.647 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:01:36.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:01:36.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:01:36.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:01:36.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:01:36.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:01:36.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:01:36.857 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:01:36.857 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:01:36.857 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:01:36.857 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:01:36.858 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:01:36.858 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:01:36.858 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:01:36.858 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:01:36.858 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:01:36.858 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:01:36.975 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:01:36.975 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:01:36.975 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:01:36.975 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:01:36.976 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:01:36.976 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:01:36.976 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:01:36.977 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:01:36.977 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:01:36.977 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:01:36.977 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:01:36.977 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:01:36.977 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:01:36.977 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:01:37.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1463: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-06-27T22:01:37.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:01:37.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:01:37.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:01:37.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:01:37.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:01:37.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:01:38.008 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:01:38.008 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:01:38.009 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:01:38.009 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:01:38.009 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:01:38.009 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:01:38.009 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:01:38.009 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:01:38.010 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:01:38.082 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1464: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:01:38.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:01:38.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:01:38.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:01:38.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:01:38.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:01:38.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:01:38.084 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:01:38.084 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:01:38.084 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:01:38.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:01:38.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:01:38.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=42a60183-a56f-4773-8c6a-cb5627517d55 2024-06-27T22:01:38.087 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 42a60183-a56f-4773-8c6a-cb5627517d55 2024-06-27T22:01:38.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 42a60183-a56f-4773-8c6a-cb5627517d55' 2024-06-27T22:01:38.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:01:38.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBC4X1m1XA9BhAA+gMdzBlE/i+5oqP00DZXhw== 2024-06-27T22:01:38.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBC4X1m1XA9BhAA+gMdzBlE/i+5oqP00DZXhw=="}' 2024-06-27T22:01:38.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 42a60183-a56f-4773-8c6a-cb5627517d55 -i td/ceph-helpers/0/new.json 2024-06-27T22:01:38.395 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:01:38.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:01:38.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBC4X1m1XA9BhAA+gMdzBlE/i+5oqP00DZXhw== --osd-uuid 42a60183-a56f-4773-8c6a-cb5627517d55 2024-06-27T22:01:38.446 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:38.443+0000 7f3c1b0767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:01:38.454 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:38.451+0000 7f3c1b0767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:01:38.456 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:38.455+0000 7f3c1b0767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:01:38.456 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:38.455+0000 7f3c1b0767c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:01:38.456 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:38.455+0000 7f3c1b0767c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:01:38.456 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:38.455+0000 7f3c1b0767c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:01:38.495 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:38.491+0000 7f3c1b0767c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:01:45.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:01:45.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:01:45.060 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:01:45.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:01:45.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:01:45.663 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:01:45.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:01:45.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:01:45.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:01:45.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:01:45.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:01:45.690 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:45.687+0000 7f10c45437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:01:45.721 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:45.719+0000 7f10c45437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:01:45.723 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:45.719+0000 7f10c45437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:01:46.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:01:46.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:01:46.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:01:46.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:01:46.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:01:46.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:46.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:01:46.119 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:01:46.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:46.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:01:46.297 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:46.295+0000 7f10c45437c0 -1 Falling back to public interface 2024-06-27T22:01:46.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:47.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:47.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:47.563 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:01:47.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:01:47.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:47.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:01:48.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:49.004 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:01:49.003+0000 7f10c45437c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:01:49.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:49.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:49.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:01:49.047 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:01:49.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:49.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:01:49.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:50.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:50.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:50.716 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:01:50.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:01:50.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:50.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:01:51.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:52.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:52.180 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:01:52.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:52.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:01:52.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:52.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:01:52.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:53.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:53.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:53.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:01:53.627 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:01:53.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:53.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:01:54.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:55.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:55.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:55.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:01:55.169 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:01:55.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:55.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:01:55.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:56.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:56.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:56.644 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:01:56.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:01:56.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:56.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:01:57.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:58.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:58.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:58.172 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:01:58.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:01:58.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:58.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:01:58.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:01:59.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:01:59.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:01:59.662 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:01:59.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:01:59.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:01:59.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:00.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:01.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:01.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:01.202 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:02:01.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:02:01.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:01.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:01.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:02.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:02.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:02.687 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:02:02.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:02:02.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:02.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:03.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:04.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:04.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:04.873 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:02:04.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:02:04.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:04.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:05.437 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:06.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:06.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:06.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:02:06.440 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:02:06.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:06.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:07.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:08.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:08.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:08.181 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:02:08.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:02:08.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:08.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:08.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:09.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:09.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:09.991 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:02:09.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:02:09.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:09.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:11.102 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:12.103 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:12.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:12.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:02:12.104 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:02:12.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:12.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:13.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:14.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:14.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:14.074 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:02:14.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:02:14.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:14.074 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:14.549 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2358868082,v1:127.0.0.1:6803/2358868082] [v2:127.0.0.1:6804/2358868082,v1:127.0.0.1:6805/2358868082] exists,up 42a60183-a56f-4773-8c6a-cb5627517d55 2024-06-27T22:02:14.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:02:14.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:02:14.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:02:14.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1465: test_get_osd_id_used_by_pgs: create_rbd_pool 2024-06-27T22:02:14.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:02:15.133 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:02:15.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:02:15.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:02:15.709 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:02:15.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:02:16.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:02:18.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1466: test_get_osd_id_used_by_pgs: wait_for_clean 2024-06-27T22:02:18.154 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:02:18.154 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:02:18.154 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:02:18.154 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:02:18.156 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:02:18.156 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:02:18.156 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:02:18.156 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:02:18.156 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:02:18.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:02:18.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:02:18.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:02:18.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:02:18.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:02:18.280 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:02:18.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:02:18.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:02:18.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:02:18.769 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:02:18.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836492 2024-06-27T22:02:18.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836492 2024-06-27T22:02:18.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492' 2024-06-27T22:02:18.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:02:18.945 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836492 2024-06-27T22:02:18.945 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:02:18.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:02:18.946 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836492 2024-06-27T22:02:18.946 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:02:18.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836492 2024-06-27T22:02:18.948 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836492 2024-06-27T22:02:18.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-06-27T22:02:18.948 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:02:19.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836492 2024-06-27T22:02:19.412 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:02:19.413 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:02:19.413 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:02:20.011 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:02:20.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:02:20.012 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:02:20.012 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:02:20.012 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:02:20.012 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:02:20.012 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:02:20.012 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:02:20.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:02:20.448 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:02:20.448 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:02:20.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:02:21.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:02:21.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:02:21.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:02:21.016 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-06-27T22:02:21.016 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: uniq 2024-06-27T22:02:21.016 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-06-27T22:02:21.017 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-06-27T22:02:21.017 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-06-27T22:02:21.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: local osd_ids=0 2024-06-27T22:02:21.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: test_get_osd_id_used_by_pgs: test 0 = 0 2024-06-27T22:02:21.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-06-27T22:02:21.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:02:21.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:02:21.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:02:21.516 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:02:21.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:02:21.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:02:21.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:02:21.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:02:21.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:02:21.625 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:02:21.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:02:21.626 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:02:21.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:02:21.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:02:21.627 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:02:21.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:02:21.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:02:21.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:02:21.628 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:02:21.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:02:21.629 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:02:21.629 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:02:21.635 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:02:21.635 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:02:21.635 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:02:21.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:02:21.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:02:21.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:02:21.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:02:21.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_osds td/ceph-helpers 2024-06-27T22:02:21.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: local dir=td/ceph-helpers 2024-06-27T22:02:21.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: setup td/ceph-helpers 2024-06-27T22:02:21.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:02:21.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:02:21.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:02:21.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:02:21.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:02:21.638 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:02:21.638 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:02:21.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:02:21.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:02:21.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:02:21.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:02:21.639 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:02:21.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:02:21.640 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:02:21.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:02:21.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:02:21.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:02:21.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:02:21.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:02:21.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:02:21.642 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:02:21.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:02:21.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:02:21.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:02:21.647 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:02:21.647 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:02:21.647 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:02:21.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:02:21.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:02:21.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:02:21.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:02:21.648 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:02:21.648 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:02:21.648 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:02:21.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:02:21.649 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:02:21.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:02:21.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:02:21.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:02:21.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-06-27T22:02:21.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:02:21.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:02:21.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:02:21.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:02:21.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:02:21.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-06-27T22:02:22.138 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:02:22.138 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:02:22.138 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:02:22.139 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:02:22.139 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:02:22.139 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:02:22.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:02:22.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-27T22:02:22.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:02:22.359 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:02:22.360 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:02:22.360 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:02:22.360 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:02:22.360 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:02:22.360 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:02:22.361 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:02:22.361 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:02:22.361 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:02:22.361 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:02:22.361 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:02:22.361 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:02:22.362 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:02:22.362 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:02:22.477 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:02:22.477 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:02:22.477 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:02:22.477 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:02:22.478 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:02:22.479 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:02:22.479 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:02:22.479 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:02:22.479 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:02:22.479 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:02:22.479 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:02:22.479 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:02:22.479 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:02:22.479 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:02:22.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: test_get_osds: run_mgr td/ceph-helpers x 2024-06-27T22:02:22.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:02:22.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:02:22.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:02:22.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:02:22.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:02:22.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:02:23.507 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:02:23.507 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:02:23.507 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:02:23.507 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:02:23.507 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:02:23.507 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:02:23.508 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:02:23.508 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:02:23.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:02:23.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: test_get_osds: run_osd td/ceph-helpers 0 2024-06-27T22:02:23.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:02:23.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:02:23.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:02:23.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:02:23.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:02:23.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:02:23.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:02:23.583 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:02:23.584 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:02:23.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:02:23.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:02:23.585 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:02:23.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a32339b2-3ffb-4ff9-9eab-890035446d73 2024-06-27T22:02:23.586 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 a32339b2-3ffb-4ff9-9eab-890035446d73 2024-06-27T22:02:23.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a32339b2-3ffb-4ff9-9eab-890035446d73' 2024-06-27T22:02:23.586 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:02:23.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBv4X1mKKYJJBAAK47fXh/GPs2gtdGscP1xaA== 2024-06-27T22:02:23.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBv4X1mKKYJJBAAK47fXh/GPs2gtdGscP1xaA=="}' 2024-06-27T22:02:23.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a32339b2-3ffb-4ff9-9eab-890035446d73 -i td/ceph-helpers/0/new.json 2024-06-27T22:02:23.928 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:02:23.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:02:23.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBv4X1mKKYJJBAAK47fXh/GPs2gtdGscP1xaA== --osd-uuid a32339b2-3ffb-4ff9-9eab-890035446d73 2024-06-27T22:02:23.981 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:23.978+0000 7f94cf4a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:02:24.005 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:24.002+0000 7f94cf4a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:02:24.007 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:24.002+0000 7f94cf4a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:02:24.008 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:24.006+0000 7f94cf4a87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:02:24.008 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:24.006+0000 7f94cf4a87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:02:24.008 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:24.006+0000 7f94cf4a87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:02:24.045 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:24.042+0000 7f94cf4a87c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:02:30.619 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:02:30.619 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:02:30.620 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:02:30.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:02:30.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:02:31.206 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:02:31.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:02:31.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:02:31.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:02:31.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:02:31.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:02:31.239 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:31.234+0000 7f0f107137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:02:31.272 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:31.270+0000 7f0f107137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:02:31.274 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:31.270+0000 7f0f107137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:02:31.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:02:31.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:02:31.678 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:02:31.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:02:31.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:02:31.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:02:31.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:31.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:02:31.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:31.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:31.853 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:31.850+0000 7f0f107137c0 -1 Falling back to public interface 2024-06-27T22:02:32.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:33.108 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:33.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:33.109 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:02:33.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:02:33.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:33.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:33.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:34.431 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:02:34.426+0000 7f0f107137c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:02:34.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:34.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:34.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:02:34.540 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:02:34.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:34.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:35.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:36.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:36.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:36.166 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:02:36.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:02:36.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:36.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:36.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:37.663 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:37.663 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:37.663 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:02:37.663 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:02:37.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:37.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:38.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:39.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:39.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:39.108 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:02:39.108 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:02:39.108 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:39.108 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:39.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:40.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:40.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:40.661 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:02:40.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:02:40.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:40.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:41.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:42.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:42.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:42.120 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:02:42.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:02:42.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:42.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:42.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:43.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:43.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:43.613 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:02:43.614 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:02:43.614 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:43.614 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:44.112 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:45.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:45.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:45.114 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:02:45.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:02:45.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:45.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:45.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:46.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:46.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:46.633 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:02:46.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:02:46.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:46.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:47.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:48.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:48.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:48.070 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:02:48.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:02:48.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:48.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:48.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:49.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:49.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:49.870 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:02:49.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:02:49.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:49.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:50.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:51.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:51.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:51.760 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:02:51.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:02:51.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:51.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:52.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:53.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:53.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:53.648 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:02:53.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:02:53.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:53.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:54.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:55.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:55.449 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:55.449 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:02:55.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:02:55.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:55.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:55.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:56.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:56.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:56.935 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:02:56.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:02:56.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:56.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:58.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:02:59.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:02:59.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:02:59.088 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:02:59.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:02:59.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:02:59.088 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:02:59.627 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3741745042,v1:127.0.0.1:6803/3741745042] [v2:127.0.0.1:6804/3741745042,v1:127.0.0.1:6805/3741745042] exists,up a32339b2-3ffb-4ff9-9eab-890035446d73 2024-06-27T22:02:59.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:02:59.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:02:59.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:02:59.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1037: test_get_osds: run_osd td/ceph-helpers 1 2024-06-27T22:02:59.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:02:59.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:02:59.631 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:02:59.632 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:02:59.632 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:02:59.632 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:02:59.632 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:02:59.632 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:02:59.633 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:02:59.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:02:59.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:02:59.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:02:59.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:02:59.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:02:59.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:02:59.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:02:59.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:02:59.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:02:59.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:02:59.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:02:59.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:02:59.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-27T22:02:59.634 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:02:59.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c5ad26b-8295-4458-bb52-d35f214c8446 2024-06-27T22:02:59.635 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 9c5ad26b-8295-4458-bb52-d35f214c8446 2024-06-27T22:02:59.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9c5ad26b-8295-4458-bb52-d35f214c8446' 2024-06-27T22:02:59.635 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:02:59.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCT4X1mtdswJxAA83q/Z/9mjsT6TB9EbQgydg== 2024-06-27T22:02:59.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCT4X1mtdswJxAA83q/Z/9mjsT6TB9EbQgydg=="}' 2024-06-27T22:02:59.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c5ad26b-8295-4458-bb52-d35f214c8446 -i td/ceph-helpers/1/new.json 2024-06-27T22:03:00.234 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:03:00.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-27T22:03:00.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCT4X1mtdswJxAA83q/Z/9mjsT6TB9EbQgydg== --osd-uuid 9c5ad26b-8295-4458-bb52-d35f214c8446 2024-06-27T22:03:00.279 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:00.274+0000 7feb06f157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:00.288 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:00.286+0000 7feb06f157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:00.289 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:00.286+0000 7feb06f157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:00.290 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:00.286+0000 7feb06f157c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:03:00.290 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:00.286+0000 7feb06f157c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:03:00.290 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:00.286+0000 7feb06f157c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:03:00.326 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:00.322+0000 7feb06f157c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-27T22:03:06.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-27T22:03:06.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:03:06.900 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:03:06.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:03:06.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:03:07.490 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:03:07.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:03:07.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:03:07.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:03:07.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:03:07.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:03:07.523 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:07.518+0000 7f468e8017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:07.561 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:07.558+0000 7f468e8017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:07.563 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:07.558+0000 7f468e8017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:07.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:03:07.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:03:07.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:03:07.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:03:07.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:03:07.947 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:03:07.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:07.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:03:07.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:07.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:08.121 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:08.118+0000 7f468e8017c0 -1 Falling back to public interface 2024-06-27T22:03:08.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:09.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:09.399 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:03:09.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:09.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:03:09.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:09.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:09.884 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:10.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:10.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:10.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:03:10.886 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:03:10.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:10.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:10.933 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:10.930+0000 7f468e8017c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:03:11.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:12.347 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:12.347 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:12.348 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:03:12.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:03:12.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:12.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:13.112 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:14.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:14.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:14.114 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:03:14.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:03:14.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:14.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:14.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:15.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:15.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:15.601 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:03:15.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:03:15.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:15.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:16.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:17.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:17.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:17.209 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:03:17.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:03:17.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:17.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:17.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:18.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:18.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:18.668 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:03:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:03:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:19.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:20.318 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:20.318 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:20.318 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:03:20.319 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:03:20.319 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:20.319 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:20.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:21.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:21.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:21.924 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:03:21.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:03:21.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:21.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:22.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:23.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:23.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:23.419 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:03:23.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:03:23.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:23.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:23.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:25.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:25.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:25.001 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:03:25.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:03:25.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:25.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:26.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:27.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:27.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:27.095 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:03:27.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:03:27.095 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:27.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:28.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:29.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:29.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:29.063 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:03:29.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:03:29.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:29.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:29.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:30.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:30.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:30.888 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:03:30.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:03:30.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:30.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:31.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:32.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:32.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:32.848 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:03:32.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:03:32.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:32.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:33.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:34.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:34.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:34.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:03:34.844 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:03:34.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:34.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:35.770 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:03:36.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:03:36.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:36.772 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:03:36.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:03:36.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:36.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:03:37.217 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3373237708,v1:127.0.0.1:6811/3373237708] [v2:127.0.0.1:6812/3373237708,v1:127.0.0.1:6813/3373237708] exists,up 9c5ad26b-8295-4458-bb52-d35f214c8446 2024-06-27T22:03:37.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:03:37.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:03:37.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:03:37.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: test_get_osds: create_rbd_pool 2024-06-27T22:03:37.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:03:37.700 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:03:37.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:03:37.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:03:38.266 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:03:38.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:03:39.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:03:41.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1039: test_get_osds: wait_for_clean 2024-06-27T22:03:41.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:03:41.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:03:41.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:03:41.389 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:03:41.390 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:03:41.390 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:03:41.390 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:03:41.390 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:03:41.390 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:03:41.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:03:41.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:03:41.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:03:41.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:03:41.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:03:41.514 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:03:41.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T22:03:41.952 INFO:tasks.workunit.client.0.smithi042.stderr:1' 2024-06-27T22:03:41.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:03:41.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:03:41.952 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:03:42.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836492 2024-06-27T22:03:42.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836492 2024-06-27T22:03:42.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492' 2024-06-27T22:03:42.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:03:42.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T22:03:42.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672969 2024-06-27T22:03:42.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672969 2024-06-27T22:03:42.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492 1-42949672969' 2024-06-27T22:03:42.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:03:42.283 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836492 2024-06-27T22:03:42.283 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:03:42.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:03:42.283 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836492 2024-06-27T22:03:42.283 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:03:42.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836492 2024-06-27T22:03:42.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-06-27T22:03:42.284 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836492 2024-06-27T22:03:42.284 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:03:42.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836492 2024-06-27T22:03:42.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:03:42.722 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672969 2024-06-27T22:03:42.722 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:03:42.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T22:03:42.724 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672969 2024-06-27T22:03:42.724 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:03:42.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672969 2024-06-27T22:03:42.725 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672969 2024-06-27T22:03:42.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-06-27T22:03:42.726 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T22:03:43.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672969 -lt 42949672969 2024-06-27T22:03:43.174 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:03:43.174 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:03:43.174 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:03:43.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:03:43.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:03:43.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:03:43.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:03:43.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:03:43.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:03:43.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:03:43.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:03:44.276 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:03:44.277 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:03:44.277 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:03:44.277 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:03:44.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:03:44.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:03:44.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:03:44.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1040: test_get_osds: create_rbd_pool 2024-06-27T22:03:44.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:03:45.387 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:03:45.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:03:45.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:03:46.246 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:03:46.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:03:47.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:03:48.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: get_osds rbd GROUP 2024-06-27T22:03:48.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=rbd 2024-06-27T22:03:48.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2024-06-27T22:03:48.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=GROUP 2024-06-27T22:03:48.986 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map rbd GROUP 2024-06-27T22:03:48.986 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-27T22:03:49.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-27T22:03:49.503 INFO:tasks.workunit.client.0.smithi042.stderr:0' 2024-06-27T22:03:49.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-27T22:03:49.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1042: test_get_osds: teardown td/ceph-helpers 2024-06-27T22:03:49.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:03:49.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:03:49.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:03:49.504 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:03:49.504 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:03:49.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:03:49.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:03:49.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:03:49.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:03:49.815 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:03:49.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:03:49.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:03:49.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:03:49.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:03:49.819 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:03:49.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:03:49.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:03:49.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:03:49.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:03:49.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:03:49.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:03:49.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:03:49.837 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:03:49.838 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:03:49.838 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:03:49.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:03:49.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:03:49.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:03:49.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:03:49.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_pg td/ceph-helpers 2024-06-27T22:03:49.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: local dir=td/ceph-helpers 2024-06-27T22:03:49.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: test_get_pg: setup td/ceph-helpers 2024-06-27T22:03:49.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:03:49.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:03:49.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:03:49.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:03:49.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:03:49.841 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:03:49.841 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:03:49.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:03:49.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:03:49.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:03:49.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:03:49.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:03:49.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:03:49.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:03:49.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:03:49.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:03:49.846 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:03:49.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:03:49.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:03:49.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:03:49.848 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:03:49.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:03:49.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:03:49.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:03:49.854 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:03:49.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:03:49.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:03:49.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:03:49.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:03:49.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:03:49.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:03:49.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:03:49.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:03:49.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:03:49.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:03:49.857 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:03:49.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:03:49.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:03:49.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:03:49.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:03:49.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:03:49.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:03:49.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:03:49.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:03:49.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:03:49.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:03:50.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:03:50.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:03:50.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:03:50.339 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:03:50.339 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:03:50.339 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:03:50.339 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:03:50.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:03:50.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:03:50.549 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:03:50.549 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:03:50.549 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:03:50.549 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:03:50.550 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:03:50.550 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:03:50.550 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:03:50.550 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:03:50.550 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:03:50.551 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:03:50.551 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:03:50.551 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:03:50.551 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:03:50.551 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:03:50.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:03:50.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:03:50.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:03:50.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:03:50.670 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:03:50.670 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:03:50.670 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:03:50.670 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:03:50.670 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:03:50.670 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:03:50.670 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:03:50.671 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:03:50.671 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:03:50.671 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:03:50.785 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: test_get_pg: run_mgr td/ceph-helpers x 2024-06-27T22:03:50.785 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:03:50.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:03:50.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:03:50.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:03:50.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:03:50.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:03:51.727 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:03:51.727 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:03:51.728 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:03:51.728 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:03:51.728 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:03:51.728 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:03:51.728 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:03:51.728 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:03:51.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:03:51.798 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: test_get_pg: run_osd td/ceph-helpers 0 2024-06-27T22:03:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:03:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:03:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:03:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:03:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:03:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:03:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:03:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:03:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:03:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:03:51.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:03:51.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:03:51.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:03:51.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:03:51.800 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:03:51.800 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:03:51.800 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:03:51.800 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:03:51.800 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:03:51.800 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:03:51.800 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:03:51.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:03:51.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:03:51.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:03:51.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=250941b0-5393-46c6-927f-7f925838cfba 2024-06-27T22:03:51.803 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 250941b0-5393-46c6-927f-7f925838cfba 2024-06-27T22:03:51.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 250941b0-5393-46c6-927f-7f925838cfba' 2024-06-27T22:03:51.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:03:51.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDH4X1m9xjsMBAAMm1opsLprZSmwDAKJ9hGWg== 2024-06-27T22:03:51.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDH4X1m9xjsMBAAMm1opsLprZSmwDAKJ9hGWg=="}' 2024-06-27T22:03:51.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 250941b0-5393-46c6-927f-7f925838cfba -i td/ceph-helpers/0/new.json 2024-06-27T22:03:52.145 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:03:52.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:03:52.163 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDH4X1m9xjsMBAAMm1opsLprZSmwDAKJ9hGWg== --osd-uuid 250941b0-5393-46c6-927f-7f925838cfba 2024-06-27T22:03:52.198 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:52.194+0000 7f79440307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:52.222 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:52.218+0000 7f79440307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:52.224 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:52.222+0000 7f79440307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:52.225 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:52.222+0000 7f79440307c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:03:52.225 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:52.222+0000 7f79440307c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:03:52.225 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:52.222+0000 7f79440307c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:03:52.262 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:52.258+0000 7f79440307c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:03:58.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:03:58.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:03:58.850 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:03:58.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:03:58.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:03:59.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:03:59.434 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:03:59.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:03:59.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:03:59.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:03:59.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:03:59.466 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:59.461+0000 7fd867ac77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:59.495 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:59.493+0000 7fd867ac77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:59.497 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:03:59.493+0000 7fd867ac77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:03:59.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:03:59.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:03:59.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:03:59.898 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:03:59.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:03:59.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:03:59.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:03:59.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:03:59.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:03:59.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:00.093 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:00.089+0000 7fd867ac77c0 -1 Falling back to public interface 2024-06-27T22:04:00.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:01.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:01.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:01.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:04:01.351 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:04:01.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:01.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:01.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:02.744 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:02.741+0000 7fd867ac77c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:04:02.795 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:02.795 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:02.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:04:02.796 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:04:02.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:02.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:03.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:04.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:04.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:04.334 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:04:04.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:04:04.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:04.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:04.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:05.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:05.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:05.806 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:04:05.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:04:05.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:05.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:06.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:07.264 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:07.264 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:07.265 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:04:07.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:04:07.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:07.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:07.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:08.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:08.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:08.806 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:04:08.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:04:08.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:08.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:09.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:10.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:10.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:10.250 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:04:10.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:04:10.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:10.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:10.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:11.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:11.868 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:04:11.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:11.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:04:11.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:11.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:12.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:13.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:13.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:13.375 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:04:13.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:04:13.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:13.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:13.910 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:14.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:14.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:14.912 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:04:14.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:04:14.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:14.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:15.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:16.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:16.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:16.411 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:04:16.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:04:16.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:16.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:17.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:18.243 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:18.243 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:18.243 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:04:18.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:04:18.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:18.244 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:19.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:20.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:20.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:20.043 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:04:20.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:04:20.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:20.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:20.892 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:21.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:21.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:21.894 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:04:21.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:04:21.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:21.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:22.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:23.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:23.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:23.723 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:04:23.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:04:23.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:23.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:24.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:25.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:25.517 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:04:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:04:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:25.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:26.663 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:27.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:27.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:27.665 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:04:27.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:04:27.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:27.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:28.170 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3305871858,v1:127.0.0.1:6803/3305871858] [v2:127.0.0.1:6804/3305871858,v1:127.0.0.1:6805/3305871858] exists,up 250941b0-5393-46c6-927f-7f925838cfba 2024-06-27T22:04:28.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:04:28.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:04:28.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:04:28.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: test_get_pg: create_rbd_pool 2024-06-27T22:04:28.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:04:28.598 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:04:28.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:04:28.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:04:29.184 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:04:29.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:04:30.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:04:31.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: test_get_pg: wait_for_clean 2024-06-27T22:04:31.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:04:31.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:04:31.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:04:31.422 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:04:31.423 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:04:31.423 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:04:31.423 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:04:31.423 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:04:31.424 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:04:31.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:04:31.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:04:31.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:04:31.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:04:31.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:04:31.550 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:04:32.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:04:32.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:04:32.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:04:32.035 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:04:32.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836494 2024-06-27T22:04:32.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836494 2024-06-27T22:04:32.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836494' 2024-06-27T22:04:32.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:04:32.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836494 2024-06-27T22:04:32.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:04:32.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:04:32.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836494 2024-06-27T22:04:32.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:04:32.215 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836494 2024-06-27T22:04:32.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836494 2024-06-27T22:04:32.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-06-27T22:04:32.215 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:04:32.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836494 2024-06-27T22:04:32.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:04:33.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:04:33.637 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:04:34.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836494 -lt 21474836494 2024-06-27T22:04:34.080 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:04:34.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:04:34.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:04:34.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:04:34.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:04:34.673 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:04:34.673 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:04:34.673 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:04:34.673 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:04:34.673 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:04:34.673 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:04:35.163 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:04:35.164 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:04:35.164 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:04:35.164 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:04:35.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:04:35.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:04:35.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:04:35.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: get_pg rbd GROUP 2024-06-27T22:04:35.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-06-27T22:04:35.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=GROUP 2024-06-27T22:04:35.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2024-06-27T22:04:35.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd GROUP 2024-06-27T22:04:35.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-27T22:04:36.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1109: test_get_pg: teardown td/ceph-helpers 2024-06-27T22:04:36.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:04:36.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:04:36.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:04:36.183 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:04:36.183 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:04:36.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:04:36.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:04:36.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:04:36.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:04:36.291 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:04:36.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:04:36.292 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:04:36.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:04:36.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:04:36.293 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:04:36.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:04:36.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:04:36.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:04:36.295 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:04:36.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:04:36.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:04:36.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:04:36.302 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:04:36.302 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:04:36.302 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:04:36.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:04:36.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:04:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:04:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:04:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_primary td/ceph-helpers 2024-06-27T22:04:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: local dir=td/ceph-helpers 2024-06-27T22:04:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: test_get_primary: setup td/ceph-helpers 2024-06-27T22:04:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:04:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:04:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:04:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:04:36.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:04:36.306 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:04:36.306 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:04:36.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:04:36.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:04:36.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:04:36.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:04:36.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:04:36.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:04:36.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:04:36.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:04:36.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:04:36.308 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:04:36.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:04:36.309 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:04:36.309 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:04:36.309 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:04:36.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:04:36.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:04:36.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:04:36.312 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:04:36.312 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:04:36.312 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:04:36.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:04:36.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:04:36.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:04:36.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:04:36.314 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:04:36.314 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:04:36.314 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:04:36.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:04:36.316 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:04:36.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:04:36.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:04:36.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:04:36.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:04:36.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:04:36.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:04:36.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:04:36.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:04:36.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:04:36.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:04:36.808 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:04:36.809 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:04:36.809 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:04:36.809 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:04:36.810 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:04:36.810 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:04:36.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:04:36.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:04:37.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:04:37.020 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:04:37.020 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:04:37.020 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:04:37.020 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:04:37.021 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:04:37.021 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:04:37.021 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:04:37.022 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:04:37.022 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:04:37.022 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:04:37.022 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:04:37.022 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:04:37.022 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:04:37.022 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:04:37.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:04:37.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:04:37.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:04:37.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:04:37.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:04:37.141 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:04:37.141 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:04:37.141 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:04:37.141 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:04:37.141 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:04:37.142 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:04:37.142 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:04:37.142 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:04:37.142 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:04:37.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: test_get_primary: local osd=0 2024-06-27T22:04:37.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: test_get_primary: run_mgr td/ceph-helpers x 2024-06-27T22:04:37.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:04:37.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:04:37.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:04:37.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:04:37.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:04:37.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:04:38.197 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:04:38.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:04:38.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:04:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:04:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:04:38.198 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:04:38.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:04:38.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:04:38.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:04:38.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: test_get_primary: run_osd td/ceph-helpers 0 2024-06-27T22:04:38.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:04:38.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:04:38.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:04:38.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:04:38.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:04:38.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:04:38.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:04:38.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:04:38.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:04:38.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:04:38.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:04:38.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:04:38.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:04:38.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:04:38.271 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:04:38.271 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:04:38.271 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:04:38.271 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:04:38.271 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:04:38.271 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:04:38.271 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:04:38.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:04:38.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:04:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:04:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:04:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:04:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:04:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:04:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:04:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:04:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:04:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:04:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:04:38.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:04:38.273 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:04:38.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14dba122-93ce-4929-b272-d9f67efaad7a 2024-06-27T22:04:38.273 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 14dba122-93ce-4929-b272-d9f67efaad7a 2024-06-27T22:04:38.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 14dba122-93ce-4929-b272-d9f67efaad7a' 2024-06-27T22:04:38.274 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:04:38.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD24X1m0i1JERAAnlHFY8TUzcgb3PYgK2nEVA== 2024-06-27T22:04:38.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD24X1m0i1JERAAnlHFY8TUzcgb3PYgK2nEVA=="}' 2024-06-27T22:04:38.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14dba122-93ce-4929-b272-d9f67efaad7a -i td/ceph-helpers/0/new.json 2024-06-27T22:04:38.615 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:04:38.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:04:38.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD24X1m0i1JERAAnlHFY8TUzcgb3PYgK2nEVA== --osd-uuid 14dba122-93ce-4929-b272-d9f67efaad7a 2024-06-27T22:04:38.668 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:38.665+0000 7f108f62f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:04:38.692 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:38.689+0000 7f108f62f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:04:38.693 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:38.689+0000 7f108f62f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:04:38.694 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:38.689+0000 7f108f62f7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:04:38.694 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:38.689+0000 7f108f62f7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:04:38.694 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:38.689+0000 7f108f62f7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:04:38.732 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:38.729+0000 7f108f62f7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:04:45.264 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:04:45.264 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:04:45.265 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:04:45.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:04:45.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:04:45.854 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:04:45.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:04:45.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:04:45.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:04:45.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:04:45.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:04:45.886 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:45.881+0000 7faf2d9157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:04:45.915 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:45.909+0000 7faf2d9157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:04:45.916 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:45.913+0000 7faf2d9157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:04:46.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:04:46.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:04:46.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:04:46.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:04:46.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:04:46.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:46.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:04:46.315 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:04:46.315 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:46.315 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:46.481 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:46.477+0000 7faf2d9157c0 -1 Falling back to public interface 2024-06-27T22:04:46.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:47.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:47.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:47.752 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:04:47.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:04:47.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:47.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:48.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:49.074 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:04:49.069+0000 7faf2d9157c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:04:49.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:49.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:49.232 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:04:49.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:04:49.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:49.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:49.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:50.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:50.816 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:04:50.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:50.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:04:50.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:50.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:51.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:52.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:52.293 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:04:52.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:52.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:04:52.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:52.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:52.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:53.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:53.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:53.757 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:04:53.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:04:53.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:53.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:54.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:55.315 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:55.315 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:55.315 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:04:55.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:04:55.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:55.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:55.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:56.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:56.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:56.783 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:04:56.784 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:04:56.784 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:56.784 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:57.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:58.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:58.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:58.327 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:04:58.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:04:58.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:58.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:04:58.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:04:59.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:04:59.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:04:59.768 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:04:59.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:04:59.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:04:59.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:00.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:01.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:01.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:01.305 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:05:01.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:05:01.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:01.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:01.747 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:02.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:02.749 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:05:02.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:02.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:05:02.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:02.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:03.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:04.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:04.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:04.468 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:05:04.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:05:04.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:04.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:05.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:06.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:06.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:06.271 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:05:06.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:05:06.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:06.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:07.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:08.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:08.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:08.152 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:05:08.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:05:08.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:08.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:08.993 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:09.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:09.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:09.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:05:09.995 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:05:09.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:09.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:11.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:12.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:12.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:12.223 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:05:12.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:05:12.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:12.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:13.107 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:14.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:14.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:14.109 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:05:14.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:05:14.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:14.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:14.602 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3117048932,v1:127.0.0.1:6803/3117048932] [v2:127.0.0.1:6804/3117048932,v1:127.0.0.1:6805/3117048932] exists,up 14dba122-93ce-4929-b272-d9f67efaad7a 2024-06-27T22:05:14.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:05:14.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:05:14.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:05:14.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: test_get_primary: create_rbd_pool 2024-06-27T22:05:14.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:05:15.122 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:05:15.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:05:15.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:05:15.823 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:05:15.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:05:16.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:05:18.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: test_get_primary: wait_for_clean 2024-06-27T22:05:18.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:05:18.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:05:18.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:05:18.173 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:05:18.174 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:18.175 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:18.175 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:18.175 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:18.175 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:18.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:05:18.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:05:18.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:05:18.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:05:18.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:05:18.299 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:05:18.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:05:18.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:05:18.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:05:18.781 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:05:18.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836484 2024-06-27T22:05:18.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836484 2024-06-27T22:05:18.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836484' 2024-06-27T22:05:18.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:05:18.957 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836484 2024-06-27T22:05:18.957 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:05:18.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:05:18.959 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836484 2024-06-27T22:05:18.959 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:05:18.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836484 2024-06-27T22:05:18.961 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836484 2024-06-27T22:05:18.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-06-27T22:05:18.961 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:05:19.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836484 -lt 21474836484 2024-06-27T22:05:19.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:05:19.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:05:19.458 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:05:20.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:05:20.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:05:20.062 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:05:20.062 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:05:20.063 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:05:20.063 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:05:20.063 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:05:20.063 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:05:20.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:05:20.511 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:05:20.511 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:05:20.511 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:05:21.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:05:21.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:05:21.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:05:21.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: get_primary rbd GROUP 2024-06-27T22:05:21.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-27T22:05:21.082 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-06-27T22:05:21.082 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-06-27T22:05:21.082 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-27T22:05:21.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: test 0 = 0 2024-06-27T22:05:21.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: test_get_primary: teardown td/ceph-helpers 2024-06-27T22:05:21.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:05:21.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:05:21.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:05:21.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:05:21.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:05:21.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:05:21.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:05:21.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:05:21.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:05:21.672 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:05:21.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:05:21.673 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:05:21.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:05:21.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:05:21.675 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:05:21.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:05:21.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:05:21.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:05:21.676 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:05:21.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:05:21.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:05:21.678 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:05:21.684 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:05:21.684 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:05:21.684 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:05:21.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:05:21.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:05:21.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:05:21.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:05:21.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_timeout_delays td/ceph-helpers 2024-06-27T22:05:21.685 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 1 2024-06-27T22:05:21.686 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:21.687 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:21.687 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:21.687 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:21.687 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:21.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '1 ' = '1 ' 2024-06-27T22:05:21.698 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 5 2024-06-27T22:05:21.698 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:21.698 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:21.699 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:21.699 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:21.699 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:21.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2024-06-27T22:05:21.719 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 6 2024-06-27T22:05:21.719 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:21.719 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:21.719 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:21.719 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:21.719 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:21.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2024-06-27T22:05:21.737 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 7 2024-06-27T22:05:21.738 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:21.738 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:21.738 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:21.738 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:21.738 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:21.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2024-06-27T22:05:21.762 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: get_timeout_delays 8 2024-06-27T22:05:21.762 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:21.762 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:21.762 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:21.762 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:21.762 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:21.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2024-06-27T22:05:21.788 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 .1 2024-06-27T22:05:21.789 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:21.789 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:21.789 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:21.790 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:21.790 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:21.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2024-06-27T22:05:21.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 1.5 .1 2024-06-27T22:05:21.817 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:21.817 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:21.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:21.818 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:21.818 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:21.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2024-06-27T22:05:21.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 5 .1 2024-06-27T22:05:21.851 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:21.851 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:21.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:21.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:21.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:21.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2024-06-27T22:05:21.894 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 6 .1 2024-06-27T22:05:21.894 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:21.894 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:21.894 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:21.894 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:21.894 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:21.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2024-06-27T22:05:21.936 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 6.3 .1 2024-06-27T22:05:21.937 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:21.937 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:21.937 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:21.937 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:21.937 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:21.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2024-06-27T22:05:21.981 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 20 .1 2024-06-27T22:05:21.981 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:21.981 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:21.981 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:21.981 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:21.981 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:22.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2024-06-27T22:05:22.040 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 300 .1 0 2024-06-27T22:05:22.040 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:22.040 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:22.041 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:22.041 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:22.041 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:22.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2024-06-27T22:05:22.131 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 300 .1 10 2024-06-27T22:05:22.133 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:05:22.133 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:05:22.133 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:05:22.133 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:05:22.133 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:05:22.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2024-06-27T22:05:22.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:05:22.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_is_clean td/ceph-helpers 2024-06-27T22:05:22.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: local dir=td/ceph-helpers 2024-06-27T22:05:22.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: test_is_clean: setup td/ceph-helpers 2024-06-27T22:05:22.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:05:22.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:05:22.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:05:22.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:05:22.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:05:22.403 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:05:22.403 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:05:22.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:05:22.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:05:22.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:05:22.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:05:22.404 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:05:22.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:05:22.404 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:05:22.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:05:22.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:05:22.406 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:05:22.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:05:22.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:05:22.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:05:22.407 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:05:22.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:05:22.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:05:22.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:05:22.410 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:05:22.411 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:05:22.411 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:05:22.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:05:22.411 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:05:22.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:05:22.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:05:22.413 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:05:22.413 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:05:22.413 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:05:22.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:05:22.415 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:05:22.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:05:22.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:05:22.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:05:22.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:05:22.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:05:22.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:05:22.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:05:22.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:05:22.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:05:22.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:05:22.899 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:05:22.899 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:05:22.899 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:05:22.899 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:05:22.900 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:05:22.900 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:05:22.900 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:05:22.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:05:23.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:05:23.120 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:05:23.120 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:05:23.120 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:05:23.120 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:05:23.121 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:05:23.121 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:05:23.122 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:05:23.122 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:05:23.122 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:05:23.122 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:05:23.122 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:05:23.122 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:05:23.122 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:05:23.122 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:05:23.243 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:05:23.243 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:05:23.243 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:05:23.243 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:05:23.244 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:05:23.245 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:05:23.245 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:05:23.245 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:05:23.245 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:05:23.245 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:05:23.245 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:05:23.245 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:05:23.245 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:05:23.245 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:05:23.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: test_is_clean: run_mgr td/ceph-helpers x 2024-06-27T22:05:23.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:05:23.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:05:23.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:05:23.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:05:23.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:05:23.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:05:24.285 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:05:24.286 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:05:24.286 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:05:24.287 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:05:24.287 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:05:24.287 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:05:24.287 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:05:24.287 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:05:24.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:05:24.350 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: test_is_clean: run_osd td/ceph-helpers 0 2024-06-27T22:05:24.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:05:24.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:05:24.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:05:24.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:05:24.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:05:24.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:05:24.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:05:24.352 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:05:24.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:05:24.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:05:24.355 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:05:24.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0b2257e0-b902-485d-af90-3add64f14ca2 2024-06-27T22:05:24.355 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 0b2257e0-b902-485d-af90-3add64f14ca2 2024-06-27T22:05:24.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0b2257e0-b902-485d-af90-3add64f14ca2' 2024-06-27T22:05:24.355 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:05:24.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAk4n1m4WU/FhAAYw7HPDjbVM80cHY4p0MxcA== 2024-06-27T22:05:24.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAk4n1m4WU/FhAAYw7HPDjbVM80cHY4p0MxcA=="}' 2024-06-27T22:05:24.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0b2257e0-b902-485d-af90-3add64f14ca2 -i td/ceph-helpers/0/new.json 2024-06-27T22:05:24.713 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:05:24.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:05:24.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAk4n1m4WU/FhAAYw7HPDjbVM80cHY4p0MxcA== --osd-uuid 0b2257e0-b902-485d-af90-3add64f14ca2 2024-06-27T22:05:24.764 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:24.761+0000 7f0461d0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:05:24.790 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:24.785+0000 7f0461d0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:05:24.791 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:24.789+0000 7f0461d0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:05:24.792 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:24.789+0000 7f0461d0e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:05:24.792 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:24.789+0000 7f0461d0e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:05:24.792 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:24.789+0000 7f0461d0e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:05:24.830 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:24.825+0000 7f0461d0e7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:05:31.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:05:31.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:05:31.289 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:05:31.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:05:31.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:05:31.906 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:05:31.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:05:31.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:05:31.907 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:05:31.907 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:05:31.907 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:05:31.939 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:31.933+0000 7f42771207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:05:31.941 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:31.937+0000 7f42771207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:05:31.943 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:31.937+0000 7f42771207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:05:32.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:05:32.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:05:32.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:05:32.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:05:32.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:05:32.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:32.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:05:32.387 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:05:32.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:32.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:32.517 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:32.513+0000 7f42771207c0 -1 Falling back to public interface 2024-06-27T22:05:32.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:33.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:33.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:33.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:05:33.880 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:05:33.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:33.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:34.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:35.204 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:05:35.201+0000 7f42771207c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:05:35.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:35.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:35.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:05:35.328 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:05:35.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:35.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:35.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:36.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:36.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:36.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:05:36.873 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:05:36.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:36.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:37.321 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:38.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:38.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:38.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:05:38.324 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:05:38.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:38.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:38.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:39.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:39.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:39.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:05:39.768 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:05:39.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:39.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:40.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:41.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:41.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:41.292 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:05:41.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:05:41.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:41.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:41.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:42.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:42.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:42.759 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:05:42.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:05:42.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:42.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:43.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:44.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:44.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:44.362 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:05:44.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:05:44.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:44.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:44.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:45.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:45.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:45.816 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:05:45.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:05:45.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:45.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:46.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:47.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:47.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:47.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:05:47.423 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:05:47.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:47.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:48.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:49.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:49.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:49.190 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:05:49.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:05:49.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:49.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:50.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:51.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:51.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:51.195 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:05:51.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:05:51.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:51.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:51.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:52.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:52.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:52.639 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:05:52.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:05:52.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:52.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:53.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:54.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:54.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:54.278 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:05:54.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:05:54.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:54.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:54.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:55.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:55.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:55.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:05:55.945 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:05:55.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:55.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:56.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:57.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:57.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:57.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:05:57.519 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:05:57.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:57.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:58.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:05:59.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:05:59.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:05:59.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:05:59.294 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:05:59.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:05:59.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:05:59.782 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3790389801,v1:127.0.0.1:6803/3790389801] [v2:127.0.0.1:6804/3790389801,v1:127.0.0.1:6805/3790389801] exists,up 0b2257e0-b902-485d-af90-3add64f14ca2 2024-06-27T22:05:59.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:05:59.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:05:59.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:05:59.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: test_is_clean: create_rbd_pool 2024-06-27T22:05:59.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:06:00.306 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:06:00.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:06:00.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:06:00.878 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:06:00.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:06:01.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:06:03.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: test_is_clean: wait_for_clean 2024-06-27T22:06:03.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:06:03.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:06:03.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:06:03.130 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:06:03.131 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:06:03.131 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:06:03.131 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:06:03.131 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:06:03.131 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:06:03.257 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:06:03.257 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:06:03.258 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:06:03.258 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:06:03.258 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:06:03.258 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:06:03.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:06:03.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:06:03.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:06:03.728 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:06:03.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836484 2024-06-27T22:06:03.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836484 2024-06-27T22:06:03.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836484' 2024-06-27T22:06:03.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:06:03.932 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836484 2024-06-27T22:06:03.932 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:06:03.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:06:03.934 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836484 2024-06-27T22:06:03.934 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:06:03.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836484 2024-06-27T22:06:03.935 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836484 2024-06-27T22:06:03.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-06-27T22:06:03.936 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:06:04.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 2 -lt 21474836484 2024-06-27T22:06:04.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:06:05.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:06:05.371 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:06:05.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836484 2024-06-27T22:06:05.864 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:06:05.864 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:06:05.864 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:06:06.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:06:06.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:06:06.526 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:06:06.526 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:06:06.526 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:06:06.526 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:06:06.526 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:06:06.526 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:06:06.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:06:06.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:06:06.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:06:06.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:06:07.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:06:07.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:06:07.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:06:07.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1569: test_is_clean: is_clean 2024-06-27T22:06:07.533 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: get_num_pgs 2024-06-27T22:06:07.534 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:06:07.534 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:06:08.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: num_pgs=4 2024-06-27T22:06:08.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: is_clean: test 4 '!=' 0 2024-06-27T22:06:08.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: get_num_active_clean 2024-06-27T22:06:08.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:06:08.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:06:08.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:06:08.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:06:08.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:06:08.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: test 4 = 4 2024-06-27T22:06:08.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: test_is_clean: teardown td/ceph-helpers 2024-06-27T22:06:08.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:06:08.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:06:08.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:06:08.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:06:08.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:06:08.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:06:08.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:06:08.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:06:08.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:06:08.712 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:06:08.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:06:08.713 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:06:08.714 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:06:08.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:06:08.715 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:06:08.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:06:08.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:06:08.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:06:08.717 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:06:08.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:06:08.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:06:08.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:06:08.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:06:08.724 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:08.724 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:08.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:06:08.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:06:08.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:06:08.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:06:08.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_kill_daemon td/ceph-helpers 2024-06-27T22:06:08.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:275: test_kill_daemon: local dir=td/ceph-helpers 2024-06-27T22:06:08.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:276: test_kill_daemon: setup td/ceph-helpers 2024-06-27T22:06:08.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:06:08.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:06:08.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:06:08.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:06:08.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:06:08.731 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:06:08.731 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:06:08.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:06:08.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:06:08.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:06:08.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:06:08.732 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:06:08.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:06:08.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:06:08.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:06:08.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:06:08.737 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:06:08.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:06:08.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:06:08.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:06:08.738 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:06:08.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:06:08.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:06:08.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:06:08.739 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:06:08.739 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:08.739 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:08.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:06:08.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:06:08.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:06:08.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:06:08.740 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:06:08.740 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:08.740 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:08.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:06:08.743 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:06:08.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:06:08.743 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:06:08.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:06:08.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:277: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:06:08.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:06:08.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:06:08.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:06:08.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:06:08.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:06:08.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:06:09.226 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:06:09.227 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:06:09.227 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:06:09.228 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:06:09.228 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:09.228 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:09.228 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:06:09.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:06:09.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:06:09.447 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:06:09.447 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:06:09.447 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:06:09.447 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:06:09.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:06:09.449 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:06:09.449 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:06:09.449 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:06:09.449 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:06:09.449 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:09.449 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:09.449 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:06:09.450 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:06:09.450 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:06:09.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:06:09.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:06:09.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:06:09.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:06:09.568 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:06:09.568 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:06:09.568 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:06:09.568 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:06:09.569 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:06:09.569 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:09.569 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:09.569 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:06:09.569 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:06:09.569 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:06:09.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:278: test_kill_daemon: run_mgr td/ceph-helpers x 2024-06-27T22:06:09.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:06:09.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:06:09.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:06:09.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:06:09.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:06:09.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:06:10.596 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:06:10.596 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:06:10.596 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:06:10.596 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:06:10.597 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:10.597 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:10.597 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:06:10.597 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:06:10.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:06:10.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:279: test_kill_daemon: run_osd td/ceph-helpers 0 2024-06-27T22:06:10.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:06:10.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:06:10.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:06:10.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:06:10.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:06:10.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:06:10.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:06:10.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:06:10.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:06:10.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:06:10.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:06:10.640 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:06:10.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4de76a8c-47ed-4be6-a689-de5142c9de7e 2024-06-27T22:06:10.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4de76a8c-47ed-4be6-a689-de5142c9de7e' 2024-06-27T22:06:10.640 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 4de76a8c-47ed-4be6-a689-de5142c9de7e 2024-06-27T22:06:10.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:06:10.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBS4n1mQnpRJxAAoGiYjwUt6poJWo7YrT1GdA== 2024-06-27T22:06:10.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBS4n1mQnpRJxAAoGiYjwUt6poJWo7YrT1GdA=="}' 2024-06-27T22:06:10.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4de76a8c-47ed-4be6-a689-de5142c9de7e -i td/ceph-helpers/0/new.json 2024-06-27T22:06:10.991 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:06:11.010 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:06:11.011 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBS4n1mQnpRJxAAoGiYjwUt6poJWo7YrT1GdA== --osd-uuid 4de76a8c-47ed-4be6-a689-de5142c9de7e 2024-06-27T22:06:11.053 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:11.048+0000 7f3e4b0757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:06:11.068 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:11.064+0000 7f3e4b0757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:06:11.070 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:11.064+0000 7f3e4b0757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:06:11.071 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:11.064+0000 7f3e4b0757c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:06:11.071 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:11.064+0000 7f3e4b0757c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:06:11.071 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:11.064+0000 7f3e4b0757c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:06:11.108 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:11.104+0000 7f3e4b0757c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:06:17.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:06:17.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:06:17.731 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:06:17.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:06:17.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:06:18.315 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:06:18.315 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:06:18.315 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:06:18.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:06:18.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:06:18.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:06:18.347 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:18.340+0000 7f600be947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:06:18.357 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:18.352+0000 7f600be947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:06:18.359 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:18.352+0000 7f600be947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:06:18.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:06:18.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:06:18.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:06:18.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:06:18.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:06:18.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:18.759 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:06:18.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:06:18.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:18.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:18.940 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:18.936+0000 7f600be947c0 -1 Falling back to public interface 2024-06-27T22:06:19.204 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:20.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:20.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:20.207 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:06:20.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:06:20.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:20.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:20.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:21.632 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:21.628+0000 7f600be947c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:06:21.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:21.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:21.644 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:06:21.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:06:21.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:21.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:22.275 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:23.276 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:23.276 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:23.277 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:06:23.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:06:23.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:23.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:23.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:24.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:24.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:24.754 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:06:24.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:06:24.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:24.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:25.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:26.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:26.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:26.206 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:06:26.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:06:26.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:26.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:26.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:27.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:27.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:27.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:06:27.796 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:06:27.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:27.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:28.238 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:29.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:29.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:29.240 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:06:29.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:06:29.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:29.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:29.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:30.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:30.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:30.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:06:30.724 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:06:30.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:30.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:31.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:32.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:32.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:32.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:06:32.181 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:06:32.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:32.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:32.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:33.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:33.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:33.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:06:33.709 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:06:33.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:33.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:34.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:35.163 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:35.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:35.164 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:06:35.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:06:35.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:35.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:35.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:36.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:36.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:36.824 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:06:36.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:06:36.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:36.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:37.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:38.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:38.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:38.895 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:06:38.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:06:38.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:38.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:39.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:41.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:41.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:41.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:06:41.000 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:06:41.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:41.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:41.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:42.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:42.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:42.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:06:42.737 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:06:42.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:42.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:43.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:44.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:44.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:44.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:06:44.690 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:06:44.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:44.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:45.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:06:46.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:06:46.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:06:46.368 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:06:46.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:06:46.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:06:46.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:06:46.809 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3670706819,v1:127.0.0.1:6803/3670706819] [v2:127.0.0.1:6804/3670706819,v1:127.0.0.1:6805/3670706819] exists,up 4de76a8c-47ed-4be6-a689-de5142c9de7e 2024-06-27T22:06:46.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:06:46.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:06:46.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:06:46.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:281: test_kill_daemon: name_prefix=osd 2024-06-27T22:06:46.811 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: grep 'osd[^/]*\.pid' 2024-06-27T22:06:46.811 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: find td/ceph-helpers 2024-06-27T22:06:46.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-06-27T22:06:46.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:289: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2024-06-27T22:06:46.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-06-27T22:06:46.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=116337 2024-06-27T22:06:46.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=0 2024-06-27T22:06:46.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local delays=1 2024-06-27T22:06:46.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-06-27T22:06:46.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-27T22:06:46.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 116337 2024-06-27T22:06:46.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-27T22:06:46.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-27T22:06:46.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2024-06-27T22:06:47.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-27T22:06:47.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 116337 2024-06-27T22:06:47.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-27T22:06:47.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-27T22:06:47.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0 2024-06-27T22:06:47.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 1 2024-06-27T22:06:47.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2024-06-27T22:06:47.819 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-06-27T22:06:47.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=116337 2024-06-27T22:06:47.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-06-27T22:06:47.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-06-27T22:06:47.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-06-27T22:06:47.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-27T22:06:47.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 116337 2024-06-27T22:06:47.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-27T22:06:47.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-27T22:06:47.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-06-27T22:06:47.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-27T22:06:47.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 116337 2024-06-27T22:06:47.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-27T22:06:47.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-27T22:06:47.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-06-27T22:06:48.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-27T22:06:48.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 116337 2024-06-27T22:06:48.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-06-27T22:06:48.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-06-27T22:06:48.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-06-27T22:06:48.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: name_prefix=mgr 2024-06-27T22:06:48.126 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: find td/ceph-helpers 2024-06-27T22:06:48.126 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: grep 'mgr[^/]*\.pid' 2024-06-27T22:06:48.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-06-27T22:06:48.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2024-06-27T22:06:48.129 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2024-06-27T22:06:48.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=115393 2024-06-27T22:06:48.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-06-27T22:06:48.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-06-27T22:06:48.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-06-27T22:06:48.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-27T22:06:48.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 115393 2024-06-27T22:06:48.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-27T22:06:48.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-27T22:06:48.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-06-27T22:06:48.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-27T22:06:48.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 115393 2024-06-27T22:06:48.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-06-27T22:06:48.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-06-27T22:06:48.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-06-27T22:06:48.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: name_prefix=mon 2024-06-27T22:06:48.234 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: find td/ceph-helpers 2024-06-27T22:06:48.235 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: grep 'mon[^/]*\.pid' 2024-06-27T22:06:48.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-06-27T22:06:48.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:309: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2024-06-27T22:06:48.238 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mon.a.pid 2024-06-27T22:06:48.239 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=115308 2024-06-27T22:06:48.239 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-06-27T22:06:48.239 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-06-27T22:06:48.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-06-27T22:06:48.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-27T22:06:48.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 115308 2024-06-27T22:06:48.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-27T22:06:48.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-27T22:06:48.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-06-27T22:06:48.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-27T22:06:48.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 115308 2024-06-27T22:06:48.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-06-27T22:06:48.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-06-27T22:06:48.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-06-27T22:06:48.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-06-27T22:06:48.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 115308 2024-06-27T22:06:48.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-06-27T22:06:48.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-06-27T22:06:48.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-06-27T22:06:48.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:310: test_kill_daemon: timeout 5 ceph status 2024-06-27T22:06:53.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:313: test_kill_daemon: teardown td/ceph-helpers 2024-06-27T22:06:53.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:06:53.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:06:53.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:06:53.548 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:06:53.548 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:06:53.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:06:53.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:06:53.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:06:53.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:06:53.557 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:06:53.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:06:53.558 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:06:53.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:06:53.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:06:53.560 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:06:53.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:06:53.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:06:53.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:06:53.561 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:06:53.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:06:53.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:06:53.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:06:53.572 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:06:53.572 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:53.572 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:53.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:06:53.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_kill_daemons td/ceph-helpers 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:366: test_kill_daemons: local dir=td/ceph-helpers 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:367: test_kill_daemons: setup td/ceph-helpers 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:06:53.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:06:53.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:06:53.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:06:53.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:06:53.577 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:06:53.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:06:53.578 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:06:53.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:06:53.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:06:53.580 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:06:53.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:06:53.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:06:53.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:06:53.581 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:06:53.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:06:53.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:06:53.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:06:53.585 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:06:53.585 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:53.585 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:53.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:06:53.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:06:53.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:06:53.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:06:53.588 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:06:53.588 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:53.588 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:53.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:06:53.591 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:06:53.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:06:53.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:06:53.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:06:53.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:06:53.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:06:53.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:06:53.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:06:53.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:06:53.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:06:53.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:06:54.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:06:54.084 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:06:54.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:06:54.085 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:06:54.085 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:54.085 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:54.085 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:06:54.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:06:54.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:06:54.305 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:06:54.305 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:06:54.305 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:06:54.305 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:06:54.306 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:06:54.307 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:06:54.307 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:06:54.307 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:06:54.307 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:06:54.307 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:54.307 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:54.307 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:06:54.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:06:54.308 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:06:54.426 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:06:54.427 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:06:54.427 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:06:54.427 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:06:54.427 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:06:54.427 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:06:54.427 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:06:54.427 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:06:54.428 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:06:54.428 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:54.428 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:54.428 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:06:54.428 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:06:54.428 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:06:54.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:369: test_kill_daemons: run_mgr td/ceph-helpers x 2024-06-27T22:06:54.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:06:54.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:06:54.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:06:54.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:06:54.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:06:54.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:06:55.461 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:06:55.461 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:06:55.462 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:06:55.462 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:06:55.462 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:55.462 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:55.463 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:06:55.463 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:06:55.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:06:55.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:370: test_kill_daemons: run_osd td/ceph-helpers 0 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:06:55.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:06:55.502 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:06:55.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:06:55.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:06:55.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:06:55.502 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:06:55.502 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:06:55.502 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:06:55.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:06:55.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:06:55.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:06:55.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:06:55.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:06:55.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:06:55.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:06:55.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:06:55.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:06:55.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:06:55.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:06:55.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:06:55.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:06:55.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:06:55.504 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:06:55.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=048ef68d-878f-49d3-bfc3-067332d5f827 2024-06-27T22:06:55.504 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 048ef68d-878f-49d3-bfc3-067332d5f827 2024-06-27T22:06:55.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 048ef68d-878f-49d3-bfc3-067332d5f827' 2024-06-27T22:06:55.505 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:06:55.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/4n1mpXQZHxAAdEl9k1Rp7xLc7J5X8FVkmQ== 2024-06-27T22:06:55.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/4n1mpXQZHxAAdEl9k1Rp7xLc7J5X8FVkmQ=="}' 2024-06-27T22:06:55.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 048ef68d-878f-49d3-bfc3-067332d5f827 -i td/ceph-helpers/0/new.json 2024-06-27T22:06:55.815 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:06:55.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:06:55.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/4n1mpXQZHxAAdEl9k1Rp7xLc7J5X8FVkmQ== --osd-uuid 048ef68d-878f-49d3-bfc3-067332d5f827 2024-06-27T22:06:55.859 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:55.856+0000 7f9804ea47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:06:55.884 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:55.880+0000 7f9804ea47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:06:55.885 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:55.880+0000 7f9804ea47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:06:55.886 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:55.880+0000 7f9804ea47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:06:55.886 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:55.880+0000 7f9804ea47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:06:55.886 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:55.880+0000 7f9804ea47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:06:55.924 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:06:55.920+0000 7f9804ea47c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:07:02.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:07:02.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:07:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:07:02.473 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:07:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:07:03.067 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:07:03.067 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:07:03.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:07:03.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:07:03.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:07:03.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:07:03.099 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:03.096+0000 7f07f3a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:07:03.132 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:03.128+0000 7f07f3a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:07:03.133 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:03.128+0000 7f07f3a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:07:03.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:07:03.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:07:03.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:07:03.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:07:03.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:07:03.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:03.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:07:03.521 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:07:03.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:03.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:03.722 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:03.716+0000 7f07f3a5c7c0 -1 Falling back to public interface 2024-06-27T22:07:03.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:04.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:04.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:04.975 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:07:04.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:07:04.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:04.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:05.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:06.253 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:06.248+0000 7f07f3a5c7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:07:06.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:06.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:06.458 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:07:06.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:07:06.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:06.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:06.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:07.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:07.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:07.974 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:07:07.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:07:07.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:07.975 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:08.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:09.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:09.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:09.507 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:07:09.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:07:09.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:09.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:09.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:10.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:10.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:10.959 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:07:10.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:07:10.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:10.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:11.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:12.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:12.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:12.486 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:07:12.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:07:12.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:12.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:12.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:13.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:13.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:13.957 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:07:13.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:07:13.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:13.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:14.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:15.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:15.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:15.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:07:15.515 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:07:15.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:15.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:15.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:16.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:16.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:16.957 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:07:16.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:07:16.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:16.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:17.486 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:18.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:18.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:18.488 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:07:18.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:07:18.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:18.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:18.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:19.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:19.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:19.934 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:07:19.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:07:19.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:19.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:20.614 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:21.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:21.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:21.616 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:07:21.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:07:21.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:21.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:22.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:23.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:23.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:23.464 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:07:23.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:07:23.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:23.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:24.356 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:25.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:25.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:25.359 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:07:25.359 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:07:25.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:25.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:26.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:27.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:27.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:27.537 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:07:27.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:07:27.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:27.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:28.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:29.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:29.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:29.393 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:07:29.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:07:29.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:29.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:30.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:31.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:31.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:31.203 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:07:31.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:07:31.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:31.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:31.649 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4130271419,v1:127.0.0.1:6803/4130271419] [v2:127.0.0.1:6804/4130271419,v1:127.0.0.1:6805/4130271419] exists,up 048ef68d-878f-49d3-bfc3-067332d5f827 2024-06-27T22:07:31.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:07:31.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:07:31.651 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:07:31.651 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:377: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2024-06-27T22:07:31.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:07:31.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:07:31.651 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:07:31.651 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:07:31.651 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:07:32.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 1 2024-06-27T22:07:32.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2024-06-27T22:07:32.660 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:07:32.660 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:07:32.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:07:32.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:07:32.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:07:32.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:07:32.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:385: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2024-06-27T22:07:32.971 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:07:32.971 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:07:32.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:07:32.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:07:32.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:07:33.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:07:33.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:389: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2024-06-27T22:07:33.080 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:07:33.080 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:07:33.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:07:33.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:07:33.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:07:33.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:07:33.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:390: test_kill_daemons: timeout 5 ceph status 2024-06-27T22:07:38.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:391: test_kill_daemons: teardown td/ceph-helpers 2024-06-27T22:07:38.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:07:38.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:07:38.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:07:38.196 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:07:38.196 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:07:38.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:07:38.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:07:38.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:07:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:07:38.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:07:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:07:38.202 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:07:38.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:07:38.204 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:07:38.204 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:07:38.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:07:38.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:07:38.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:07:38.206 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:07:38.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:07:38.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:07:38.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:07:38.216 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:07:38.217 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:07:38.217 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:07:38.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:07:38.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:07:38.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:07:38.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:07:38.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_objectstore_tool td/ceph-helpers 2024-06-27T22:07:38.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: local dir=td/ceph-helpers 2024-06-27T22:07:38.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: setup td/ceph-helpers 2024-06-27T22:07:38.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:07:38.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:07:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:07:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:07:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:07:38.221 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:07:38.221 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:07:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:07:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:07:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:07:38.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:07:38.222 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:07:38.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:07:38.223 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:07:38.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:07:38.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:07:38.225 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:07:38.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:07:38.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:07:38.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:07:38.227 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:07:38.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:07:38.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:07:38.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:07:38.231 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:07:38.231 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:07:38.231 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:07:38.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:07:38.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:07:38.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:07:38.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:07:38.234 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:07:38.234 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:07:38.234 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:07:38.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:07:38.236 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:07:38.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:07:38.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:07:38.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:07:38.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:07:38.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:07:38.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:07:38.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:07:38.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:07:38.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:07:38.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:07:38.692 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:07:38.692 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:07:38.692 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:07:38.693 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:07:38.693 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:07:38.693 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:07:38.693 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:07:38.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:07:38.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:07:38.913 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:07:38.913 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:07:38.913 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:07:38.914 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:07:38.914 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:07:38.914 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:07:38.914 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:07:38.914 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:07:38.915 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:07:38.915 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:07:38.915 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:07:38.915 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:07:38.916 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:07:38.916 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:07:39.034 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:07:39.034 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:07:39.035 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:07:39.035 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:07:39.036 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:07:39.036 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:07:39.036 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:07:39.036 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:07:39.036 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:07:39.036 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:07:39.036 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:07:39.036 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:07:39.037 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:07:39.037 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:07:39.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: local osd=0 2024-06-27T22:07:39.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: test_objectstore_tool: run_mgr td/ceph-helpers x 2024-06-27T22:07:39.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:07:39.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:07:39.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:07:39.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:07:39.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:07:39.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:07:40.074 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:07:40.074 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:07:40.074 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:07:40.075 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:07:40.075 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:07:40.075 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:07:40.075 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:07:40.075 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:07:40.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:07:40.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: test_objectstore_tool: run_osd td/ceph-helpers 0 2024-06-27T22:07:40.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:07:40.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:07:40.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:07:40.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:07:40.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:07:40.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:07:40.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:07:40.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:07:40.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:07:40.147 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:07:40.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:07:40.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:07:40.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:07:40.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:07:40.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:07:40.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:07:40.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:07:40.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:07:40.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:07:40.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:07:40.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:07:40.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:07:40.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:07:40.149 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:07:40.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aaa46a36-a372-4dec-a3fd-5de258c92c27 2024-06-27T22:07:40.150 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 aaa46a36-a372-4dec-a3fd-5de258c92c27 2024-06-27T22:07:40.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 aaa46a36-a372-4dec-a3fd-5de258c92c27' 2024-06-27T22:07:40.151 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:07:40.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCs4n1mD0beCRAAMyBU/f9R/RBVYsFQWK1n6Q== 2024-06-27T22:07:40.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCs4n1mD0beCRAAMyBU/f9R/RBVYsFQWK1n6Q=="}' 2024-06-27T22:07:40.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aaa46a36-a372-4dec-a3fd-5de258c92c27 -i td/ceph-helpers/0/new.json 2024-06-27T22:07:40.475 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:07:40.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:07:40.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCs4n1mD0beCRAAMyBU/f9R/RBVYsFQWK1n6Q== --osd-uuid aaa46a36-a372-4dec-a3fd-5de258c92c27 2024-06-27T22:07:40.521 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:40.520+0000 7f0aa41c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:07:40.543 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:40.540+0000 7f0aa41c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:07:40.544 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:40.544+0000 7f0aa41c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:07:40.545 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:40.544+0000 7f0aa41c17c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:07:40.545 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:40.544+0000 7f0aa41c17c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:07:40.545 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:40.544+0000 7f0aa41c17c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:07:40.582 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:40.580+0000 7f0aa41c17c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:07:47.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:07:47.005 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:07:47.007 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:07:47.007 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:07:47.007 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:07:47.587 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:07:47.587 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:07:47.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:07:47.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:07:47.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:07:47.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:07:47.620 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:47.620+0000 7ff116dd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:07:47.622 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:47.620+0000 7ff116dd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:07:47.623 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:47.624+0000 7ff116dd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:07:48.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:07:48.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:07:48.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:07:48.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:07:48.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:07:48.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:48.063 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:07:48.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:07:48.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:48.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:48.184 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:48.184+0000 7ff116dd17c0 -1 Falling back to public interface 2024-06-27T22:07:48.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:49.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:49.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:49.555 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:07:49.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:07:49.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:49.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:50.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:50.823 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:07:50.820+0000 7ff116dd17c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:07:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:51.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:51.057 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:07:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:07:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:51.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:52.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:52.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:52.649 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:07:52.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:07:52.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:52.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:53.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:54.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:54.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:54.098 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:07:54.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:07:54.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:54.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:54.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:55.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:55.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:55.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:07:55.642 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:07:55.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:55.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:56.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:57.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:57.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:57.122 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:07:57.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:07:57.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:57.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:57.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:07:58.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:07:58.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:07:58.673 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:07:58.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:07:58.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:07:58.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:07:59.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:00.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:00.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:00.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:08:00.137 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:08:00.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:00.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:00.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:01.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:01.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:01.668 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:08:01.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:08:01.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:01.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:02.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:03.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:03.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:03.174 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:08:03.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:08:03.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:03.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:03.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:04.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:04.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:04.709 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:08:04.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:08:04.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:04.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:05.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:06.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:06.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:06.287 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:08:06.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:08:06.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:06.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:07.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:08.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:08.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:08.035 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:08:08.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:08:08.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:08.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:09.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:10.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:10.161 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:08:10.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:10.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:08:10.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:10.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:11.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:12.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:12.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:12.040 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:08:12.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:08:12.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:12.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:12.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:13.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:13.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:13.872 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:08:13.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:08:13.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:13.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:14.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:15.826 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:15.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:15.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:08:15.827 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:08:15.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:15.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:16.313 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1568413289,v1:127.0.0.1:6803/1568413289] [v2:127.0.0.1:6804/1568413289,v1:127.0.0.1:6805/1568413289] exists,up aaa46a36-a372-4dec-a3fd-5de258c92c27 2024-06-27T22:08:16.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:08:16.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:08:16.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:08:16.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: test_objectstore_tool: create_rbd_pool 2024-06-27T22:08:16.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:08:16.761 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:08:16.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:08:16.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:08:17.380 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:08:17.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:08:18.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:08:19.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: test_objectstore_tool: wait_for_clean 2024-06-27T22:08:19.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:08:19.629 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:08:19.629 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:08:19.629 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:08:19.630 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:08:19.630 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:08:19.631 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:08:19.631 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:08:19.631 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:08:19.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:08:19.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:08:19.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:08:19.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:08:19.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:08:19.757 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:08:20.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:08:20.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:08:20.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:08:20.227 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:08:20.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836487 2024-06-27T22:08:20.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836487 2024-06-27T22:08:20.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487' 2024-06-27T22:08:20.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:08:20.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836487 2024-06-27T22:08:20.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:08:20.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:08:20.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-06-27T22:08:20.402 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:08:20.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836487 2024-06-27T22:08:20.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-06-27T22:08:20.403 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836487 2024-06-27T22:08:20.403 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:08:20.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836484 -lt 21474836487 2024-06-27T22:08:20.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:08:21.826 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:08:21.827 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:08:22.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836487 2024-06-27T22:08:22.267 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:08:22.268 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:08:22.268 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:08:22.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:08:22.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:08:22.857 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:08:22.857 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:08:22.857 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:08:22.857 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:08:22.857 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:08:22.857 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:08:23.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:08:23.302 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:08:23.303 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:08:23.303 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:08:23.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:08:23.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:08:23.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:08:23.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1316: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2024-06-27T22:08:24.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2024-06-27T22:08:24.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: test_objectstore_tool: diff - /etc/group 2024-06-27T22:08:24.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-06-27T22:08:24.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-27T22:08:24.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-27T22:08:24.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-27T22:08:24.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2024-06-27T22:08:24.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-06-27T22:08:24.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-27T22:08:24.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-27T22:08:24.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-27T22:08:24.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-06-27T22:08:24.069 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:08:24.069 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:08:24.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:08:24.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:08:24.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:08:24.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:08:24.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2024-06-27T22:08:24.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-06-27T22:08:24.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-27T22:08:24.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-27T22:08:24.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-27T22:08:24.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-06-27T22:08:24.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2024-06-27T22:08:24.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2024-06-27T22:08:24.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-27T22:08:24.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-27T22:08:24.945 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:08:24.946 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:08:24.947 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:08:24.947 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:08:24.947 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:08:24.947 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:08:24.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:08:24.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:08:24.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:08:24.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:08:24.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:08:24.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:08:24.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:08:24.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:08:24.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-27T22:08:24.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-27T22:08:24.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:08:24.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-27T22:08:24.949 INFO:tasks.workunit.client.0.smithi042.stderr:start osd.0 2024-06-27T22:08:24.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:08:24.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-06-27T22:08:24.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-27T22:08:24.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-27T22:08:24.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-27T22:08:24.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-27T22:08:24.983 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:24.980+0000 7f17a502c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:08:24.986 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:24.984+0000 7f17a502c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:08:24.988 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:24.988+0000 7f17a502c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:08:25.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-27T22:08:25.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:08:25.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:08:25.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:08:25.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:08:25.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:25.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:08:25.394 INFO:tasks.workunit.client.0.smithi042.stderr:0 2024-06-27T22:08:25.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:25.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:25.561 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:25.560+0000 7f17a502c7c0 -1 Falling back to public interface 2024-06-27T22:08:25.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:26.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:26.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:26.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:08:26.883 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T22:08:26.884 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:26.884 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:27.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:28.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:28.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:28.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:08:28.343 INFO:tasks.workunit.client.0.smithi042.stderr:2 2024-06-27T22:08:28.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:28.344 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:28.601 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:28.600+0000 7f17a502c7c0 -1 osd.0 12 log_to_monitors true 2024-06-27T22:08:28.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:29.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:29.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:29.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:08:29.840 INFO:tasks.workunit.client.0.smithi042.stderr:3 2024-06-27T22:08:29.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:29.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:30.289 INFO:tasks.workunit.client.0.smithi042.stderr:osd.0 up in weight 1 up_from 16 up_thru 16 down_at 13 last_clean_interval [5,12) [v2:127.0.0.1:6802/2995683155,v1:127.0.0.1:6803/2995683155] [v2:127.0.0.1:6804/2995683155,v1:127.0.0.1:6805/2995683155] exists,up aaa46a36-a372-4dec-a3fd-5de258c92c27 2024-06-27T22:08:30.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:08:30.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:08:30.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:08:30.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-27T22:08:30.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:08:30.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:08:30.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:08:30.290 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:08:30.290 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:08:30.290 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:08:30.291 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:08:30.291 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:08:30.291 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:08:30.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:08:30.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:08:30.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:08:30.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:08:30.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:08:30.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:08:30.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:08:30.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:08:30.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:08:30.896 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:08:31.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=68719476739 2024-06-27T22:08:31.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 68719476739 2024-06-27T22:08:31.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-68719476739' 2024-06-27T22:08:31.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:08:31.065 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-68719476739 2024-06-27T22:08:31.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:08:31.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:08:31.067 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-68719476739 2024-06-27T22:08:31.067 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:08:31.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=68719476739 2024-06-27T22:08:31.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 68719476739' 2024-06-27T22:08:31.069 INFO:tasks.workunit.client.0.smithi042.stderr:waiting osd.0 seq 68719476739 2024-06-27T22:08:31.069 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:08:31.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 68719476739 -lt 68719476739 2024-06-27T22:08:31.511 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:08:31.512 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:08:31.512 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:08:32.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:08:32.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:08:32.088 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:08:32.088 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:08:32.088 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:08:32.088 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:08:32.088 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:08:32.089 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:08:32.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:08:32.524 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:08:32.524 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:08:32.524 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:08:33.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:08:33.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:08:33.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:08:33.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1319: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2024-06-27T22:08:33.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-06-27T22:08:33.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-27T22:08:33.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-27T22:08:33.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-27T22:08:33.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2024-06-27T22:08:33.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-06-27T22:08:33.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-27T22:08:33.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-27T22:08:33.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-27T22:08:33.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-06-27T22:08:33.094 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:08:33.094 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:08:33.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:08:33.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:08:33.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:08:33.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:08:33.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2024-06-27T22:08:33.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-06-27T22:08:33.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-27T22:08:33.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-27T22:08:33.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-27T22:08:33.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-06-27T22:08:33.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2024-06-27T22:08:33.499 INFO:tasks.workunit.client.0.smithi042.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2024-06-27T22:08:33.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-06-27T22:08:33.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: return 1 2024-06-27T22:08:33.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: return 1 2024-06-27T22:08:33.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1320: test_objectstore_tool: teardown td/ceph-helpers 2024-06-27T22:08:33.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:08:33.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:08:33.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:08:33.786 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:08:33.786 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:08:33.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:08:33.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:08:33.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:08:33.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:08:33.895 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:08:33.896 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:08:33.896 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:08:33.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:08:33.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:08:33.898 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:08:33.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:08:33.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:08:33.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:08:33.899 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:08:33.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:08:33.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:08:33.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:08:33.909 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:08:33.909 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:08:33.909 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:08:33.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_pg_schedule_scrub td/ceph-helpers 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: test_pg_schedule_scrub: local dir=td/ceph-helpers 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: test_pg_schedule_scrub: setup td/ceph-helpers 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:08:33.912 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:08:33.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:08:33.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:08:33.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:08:33.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:08:33.914 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:08:33.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:08:33.915 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:08:33.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:08:33.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:08:33.916 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:08:33.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:08:33.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:08:33.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:08:33.918 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:08:33.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:08:33.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:08:33.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:08:33.921 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:08:33.921 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:08:33.921 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:08:33.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:08:33.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:08:33.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:08:33.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:08:33.923 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:08:33.923 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:08:33.923 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:08:33.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:08:33.924 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:08:33.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:08:33.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:08:33.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:08:33.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1950: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:08:33.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:08:33.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:08:33.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:08:33.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:08:33.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:08:33.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:08:34.422 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:08:34.422 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:08:34.422 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:08:34.423 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:08:34.423 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:08:34.423 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:08:34.423 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:08:34.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:08:34.692 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:08:34.692 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:08:34.693 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:08:34.693 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:08:34.693 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:08:34.694 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:08:34.694 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:08:34.694 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:08:34.694 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:08:34.694 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:08:34.694 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:08:34.694 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:08:34.695 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:08:34.695 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:08:34.695 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:08:34.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:08:34.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:08:34.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:08:34.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:08:34.814 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:08:34.814 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:08:34.814 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:08:34.814 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:08:34.815 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:08:34.815 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:08:34.815 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:08:34.815 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:08:34.815 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:08:34.815 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:08:34.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: test_pg_schedule_scrub: run_mgr td/ceph-helpers x 2024-06-27T22:08:34.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:08:34.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:08:34.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:08:34.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:08:34.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:08:34.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:08:35.832 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:08:35.832 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:08:35.833 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:08:35.833 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:08:35.833 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:08:35.833 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:08:35.833 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:08:35.834 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:08:35.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:08:35.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1952: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2024-06-27T22:08:35.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:08:35.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:08:35.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:08:35.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:08:35.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:08:35.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:08:35.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:08:35.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:08:35.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:08:35.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:08:35.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:08:35.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:08:35.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:08:35.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:08:35.876 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:08:35.876 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:08:35.876 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:08:35.877 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:08:35.877 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:08:35.877 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:08:35.877 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:08:35.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:08:35.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:08:35.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:08:35.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:08:35.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:08:35.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:08:35.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:08:35.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:08:35.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:08:35.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:08:35.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:08:35.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:08:35.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:08:35.880 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:08:35.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=27c896bb-008a-42f5-8437-7e6924d3d7d9 2024-06-27T22:08:35.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 27c896bb-008a-42f5-8437-7e6924d3d7d9' 2024-06-27T22:08:35.881 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 27c896bb-008a-42f5-8437-7e6924d3d7d9 2024-06-27T22:08:35.881 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:08:35.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDj4n1m5n0CNRAA8oiK5YKm5lyt9EDt2d9KKQ== 2024-06-27T22:08:35.890 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDj4n1m5n0CNRAA8oiK5YKm5lyt9EDt2d9KKQ=="}' 2024-06-27T22:08:35.890 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 27c896bb-008a-42f5-8437-7e6924d3d7d9 -i td/ceph-helpers/0/new.json 2024-06-27T22:08:36.211 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:08:36.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:08:36.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDj4n1m5n0CNRAA8oiK5YKm5lyt9EDt2d9KKQ== --osd-uuid 27c896bb-008a-42f5-8437-7e6924d3d7d9 2024-06-27T22:08:36.270 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:36.268+0000 7fae391f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:08:36.298 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:36.296+0000 7fae391f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:08:36.299 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:36.300+0000 7fae391f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:08:36.300 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:36.300+0000 7fae391f87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:08:36.300 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:36.300+0000 7fae391f87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:08:36.300 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:36.300+0000 7fae391f87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:08:36.337 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:36.336+0000 7fae391f87c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:08:42.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:08:42.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:08:42.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:08:42.900 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:08:42.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:08:43.534 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:08:43.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:08:43.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:08:43.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:08:43.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:08:43.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:08:43.567 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:43.567+0000 7f81233077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:08:43.570 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:43.567+0000 7f81233077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:08:43.571 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:43.571+0000 7f81233077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:08:43.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:08:43.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:08:43.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:08:43.957 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:08:43.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:08:43.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:08:43.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:43.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:08:43.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:43.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:44.137 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:44.135+0000 7f81233077c0 -1 Falling back to public interface 2024-06-27T22:08:44.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:45.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:45.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:45.398 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:08:45.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:08:45.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:45.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:45.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:46.795 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:08:46.791+0000 7f81233077c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:08:46.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:46.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:46.832 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:08:46.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:08:46.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:46.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:47.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:48.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:48.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:48.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:08:48.434 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:08:48.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:48.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:48.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:49.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:49.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:49.873 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:08:49.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:08:49.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:49.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:50.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:51.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:51.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:51.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:08:51.367 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:08:51.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:51.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:51.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:52.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:52.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:52.905 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:08:52.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:08:52.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:52.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:53.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:54.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:54.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:54.353 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:08:54.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:08:54.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:54.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:55.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:56.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:56.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:56.003 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:08:56.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:08:56.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:56.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:56.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:57.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:57.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:57.595 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:08:57.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:08:57.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:57.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:58.089 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:08:59.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:08:59.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:08:59.092 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:08:59.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:08:59.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:08:59.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:08:59.564 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:00.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:00.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:00.566 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:09:00.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:09:00.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:00.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:01.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:02.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:02.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:02.521 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:09:02.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:09:02.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:02.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:03.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:04.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:04.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:04.457 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:09:04.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:09:04.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:04.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:05.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:06.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:06.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:06.351 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:09:06.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:09:06.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:06.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:07.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:08.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:08.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:08.348 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:09:08.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:09:08.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:08.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:09.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:10.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:10.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:10.525 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:09:10.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:09:10.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:10.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:11.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:12.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:12.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:12.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:09:12.128 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:09:12.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:12.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:12.568 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3539388952,v1:127.0.0.1:6803/3539388952] [v2:127.0.0.1:6804/3539388952,v1:127.0.0.1:6805/3539388952] exists,up 27c896bb-008a-42f5-8437-7e6924d3d7d9 2024-06-27T22:09:12.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:09:12.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:09:12.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:09:12.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1953: test_pg_schedule_scrub: create_rbd_pool 2024-06-27T22:09:12.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:09:13.032 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:09:13.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:09:13.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:09:13.649 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:09:13.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:09:14.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:09:15.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: test_pg_schedule_scrub: wait_for_clean 2024-06-27T22:09:15.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:09:15.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:09:15.890 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:09:15.890 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:09:15.891 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:09:15.891 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:09:15.891 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:09:15.891 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:09:15.892 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:09:16.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:09:16.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:09:16.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:09:16.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:09:16.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:09:16.017 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:09:16.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:09:16.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:09:16.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:09:16.506 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:09:16.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836490 2024-06-27T22:09:16.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836490 2024-06-27T22:09:16.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836490' 2024-06-27T22:09:16.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:09:16.701 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:09:16.701 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836490 2024-06-27T22:09:16.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:09:16.702 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836490 2024-06-27T22:09:16.703 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:09:16.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836490 2024-06-27T22:09:16.704 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836490 2024-06-27T22:09:16.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-06-27T22:09:16.704 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:09:17.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836490 -lt 21474836490 2024-06-27T22:09:17.180 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:09:17.181 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:09:17.181 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:09:17.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:09:17.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:09:17.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:09:17.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:09:17.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:09:17.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:09:17.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:09:17.811 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:09:18.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:09:18.261 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:09:18.262 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:09:18.262 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:09:18.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:09:18.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:09:18.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:09:18.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1955: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2024-06-27T22:09:18.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: pg_schedule_scrub: local pgid=1.0 2024-06-27T22:09:18.847 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_schedule_scrub: get_last_scrub_stamp 1.0 2024-06-27T22:09:18.847 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:09:18.847 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:09:18.847 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:09:18.847 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:09:19.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_schedule_scrub: local last_scrub=2024-06-27T22:09:13.474954+0000 2024-06-27T22:09:19.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_schedule_scrub: ceph pg scrub 1.0 2024-06-27T22:09:19.699 INFO:tasks.workunit.client.0.smithi042.stderr:instructing pg 1.0 on osd.0 to scrub 2024-06-27T22:09:19.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_schedule_scrub: wait_for_scrub 1.0 2024-06-27T22:09:13.474954+0000 2024-06-27T22:09:19.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-27T22:09:19.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-27T22:09:13.474954+0000 2024-06-27T22:09:19.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-27T22:09:19.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-27T22:09:19.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:09:19.717 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:09:19.717 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:09:19.717 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:09:19.717 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:09:19.717 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:09:20.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:09:13.474954+0000 '>' 2024-06-27T22:09:13.474954+0000 2024-06-27T22:09:20.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:09:21.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:09:21.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:09:21.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:09:21.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:09:21.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:09:21.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:09:21.140 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:09:21.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:09:13.474954+0000 '>' 2024-06-27T22:09:13.474954+0000 2024-06-27T22:09:21.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:09:22.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:09:22.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:09:22.579 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:09:22.579 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:09:22.579 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:09:22.579 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:09:22.579 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:09:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:09:19.806667+0000 '>' 2024-06-27T22:09:13.474954+0000 2024-06-27T22:09:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-27T22:09:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: test_pg_schedule_scrub: kill_daemons td/ceph-helpers KILL osd 2024-06-27T22:09:23.009 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:09:23.009 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:09:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:09:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:09:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:09:23.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:09:23.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1957: test_pg_schedule_scrub: TIMEOUT=1 2024-06-27T22:09:23.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1957: test_pg_schedule_scrub: pg_scrub 1.0 2024-06-27T22:09:23.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: local pgid=1.0 2024-06-27T22:09:23.118 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: get_last_scrub_stamp 1.0 2024-06-27T22:09:23.118 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:09:23.118 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:09:23.118 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:09:23.118 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:09:23.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local last_scrub=2024-06-27T22:09:19.806667+0000 2024-06-27T22:09:23.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: ceph pg scrub 1.0 2024-06-27T22:09:23.947 INFO:tasks.workunit.client.0.smithi042.stderr:instructing pg 1.0 on osd.0 to scrub 2024-06-27T22:09:23.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: wait_for_scrub 1.0 2024-06-27T22:09:19.806667+0000 2024-06-27T22:09:23.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-27T22:09:23.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-27T22:09:19.806667+0000 2024-06-27T22:09:23.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-27T22:09:23.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-27T22:09:23.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-06-27T22:09:23.964 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:09:23.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:09:23.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:09:23.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:09:23.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:09:24.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:09:19.806667+0000 '>' 2024-06-27T22:09:19.806667+0000 2024-06-27T22:09:24.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:09:25.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:09:25.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-06-27T22:09:25.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: wait_for_scrub: return 1 2024-06-27T22:09:25.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1958: test_pg_schedule_scrub: teardown td/ceph-helpers 2024-06-27T22:09:25.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:09:25.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:09:25.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:09:25.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:09:25.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:09:25.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:09:25.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:09:25.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:09:25.502 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:09:25.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:09:25.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:09:25.503 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:09:25.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:09:25.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:09:25.504 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:09:25.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:09:25.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:09:25.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:09:25.506 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:09:25.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:09:25.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:09:25.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:09:25.514 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:09:25.515 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:09:25.515 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:09:25.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:09:25.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_pg_scrub td/ceph-helpers 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1905: test_pg_scrub: local dir=td/ceph-helpers 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: test_pg_scrub: setup td/ceph-helpers 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:09:25.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:09:25.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:09:25.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:09:25.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:09:25.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:09:25.518 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:09:25.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:09:25.519 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:09:25.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:09:25.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:09:25.521 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:09:25.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:09:25.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:09:25.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:09:25.522 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:09:25.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:09:25.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:09:25.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:09:25.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:09:25.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:09:25.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:09:25.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:09:25.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:09:25.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:09:25.526 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:09:25.527 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:09:25.527 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:09:25.527 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:09:25.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:09:25.529 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:09:25.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:09:25.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:09:25.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:09:25.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:09:25.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:09:25.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:09:25.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:09:25.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:09:25.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:09:25.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:09:26.024 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:09:26.024 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:09:26.024 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:09:26.025 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:09:26.025 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:09:26.025 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:09:26.025 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:09:26.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:09:26.243 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:09:26.244 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:09:26.245 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:09:26.245 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:09:26.245 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:09:26.245 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:09:26.246 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:09:26.246 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:09:26.246 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:09:26.247 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:09:26.247 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:09:26.247 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:09:26.247 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:09:26.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:09:26.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:09:26.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:09:26.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:09:26.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:09:26.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:09:26.364 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:09:26.364 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:09:26.364 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:09:26.364 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:09:26.364 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:09:26.365 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:09:26.365 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:09:26.365 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:09:26.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:09:26.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:09:26.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1909: test_pg_scrub: run_mgr td/ceph-helpers x 2024-06-27T22:09:26.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:09:26.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:09:26.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:09:26.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:09:26.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:09:26.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:09:27.403 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:09:27.403 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:09:27.403 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:09:27.403 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:09:27.403 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:09:27.403 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:09:27.403 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:09:27.404 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:09:27.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:09:27.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: test_pg_scrub: run_osd td/ceph-helpers 0 2024-06-27T22:09:27.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:09:27.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:09:27.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:09:27.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:09:27.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:09:27.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:09:27.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:09:27.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:09:27.442 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:09:27.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:09:27.444 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:09:27.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=64b5a1d1-2533-46db-96c0-5572bec7d2b0 2024-06-27T22:09:27.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 64b5a1d1-2533-46db-96c0-5572bec7d2b0' 2024-06-27T22:09:27.445 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 64b5a1d1-2533-46db-96c0-5572bec7d2b0 2024-06-27T22:09:27.445 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:09:27.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAX431mKqWBGxAAp/Il0KjZ614sehRIRr3u9A== 2024-06-27T22:09:27.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAX431mKqWBGxAAp/Il0KjZ614sehRIRr3u9A=="}' 2024-06-27T22:09:27.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 64b5a1d1-2533-46db-96c0-5572bec7d2b0 -i td/ceph-helpers/0/new.json 2024-06-27T22:09:27.806 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:09:27.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:09:27.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAX431mKqWBGxAAp/Il0KjZ614sehRIRr3u9A== --osd-uuid 64b5a1d1-2533-46db-96c0-5572bec7d2b0 2024-06-27T22:09:27.858 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:27.855+0000 7f4b710167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:09:27.883 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:27.879+0000 7f4b710167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:09:27.884 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:27.883+0000 7f4b710167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:09:27.885 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:27.883+0000 7f4b710167c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:09:27.885 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:27.883+0000 7f4b710167c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:09:27.885 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:27.883+0000 7f4b710167c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:09:27.923 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:27.919+0000 7f4b710167c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:09:34.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:09:34.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:09:34.506 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:09:34.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:09:34.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:09:35.118 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:09:35.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:09:35.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:09:35.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:09:35.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:09:35.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:09:35.152 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:35.151+0000 7f2a8ed237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:09:35.166 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:35.163+0000 7f2a8ed237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:09:35.169 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:35.167+0000 7f2a8ed237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:09:35.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:09:35.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:09:35.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:09:35.552 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:09:35.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:09:35.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:09:35.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:35.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:09:35.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:35.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:35.756 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:35.755+0000 7f2a8ed237c0 -1 Falling back to public interface 2024-06-27T22:09:36.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:37.032 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:37.032 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:37.032 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:09:37.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:09:37.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:37.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:37.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:38.308 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:09:38.307+0000 7f2a8ed237c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:09:38.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:38.520 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:09:38.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:38.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:09:38.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:38.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:39.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:40.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:40.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:40.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:09:40.160 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:09:40.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:40.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:40.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:41.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:41.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:41.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:09:41.641 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:09:41.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:41.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:42.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:43.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:43.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:43.170 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:09:43.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:09:43.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:43.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:43.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:44.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:44.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:44.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:09:44.621 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:09:44.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:44.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:45.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:46.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:46.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:46.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:09:46.181 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:09:46.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:46.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:46.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:47.699 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:47.699 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:47.699 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:09:47.699 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:09:47.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:47.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:48.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:49.264 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:49.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:49.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:09:49.265 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:09:49.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:49.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:49.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:50.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:50.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:50.709 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:09:50.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:09:50.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:50.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:51.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:52.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:52.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:52.400 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:09:52.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:09:52.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:52.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:53.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:54.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:54.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:54.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:09:54.393 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:09:54.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:54.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:54.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:55.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:55.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:55.845 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:09:55.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:09:55.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:55.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:56.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:57.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:57.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:57.673 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:09:57.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:09:57.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:57.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:09:58.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:09:59.502 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:09:59.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:09:59.503 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:09:59.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:09:59.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:09:59.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:00.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:01.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:01.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:01.541 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:10:01.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:10:01.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:01.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:02.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:03.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:03.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:03.182 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:10:03.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:10:03.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:03.182 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:03.626 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1649456856,v1:127.0.0.1:6803/1649456856] [v2:127.0.0.1:6804/1649456856,v1:127.0.0.1:6805/1649456856] exists,up 64b5a1d1-2533-46db-96c0-5572bec7d2b0 2024-06-27T22:10:03.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:10:03.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:10:03.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:10:03.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: test_pg_scrub: create_rbd_pool 2024-06-27T22:10:03.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:10:04.103 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:10:04.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:10:04.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:10:04.693 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:10:04.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:10:05.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:10:06.884 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: test_pg_scrub: wait_for_clean 2024-06-27T22:10:06.884 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:10:06.884 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:10:06.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:10:06.885 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:10:06.886 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:10:06.886 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:10:06.887 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:10:06.887 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:10:06.887 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:10:07.006 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:10:07.006 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:10:07.006 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:10:07.007 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:10:07.007 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:10:07.007 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:10:07.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:10:07.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:10:07.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:10:07.441 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:10:07.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836487 2024-06-27T22:10:07.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836487 2024-06-27T22:10:07.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487' 2024-06-27T22:10:07.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:10:07.624 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836487 2024-06-27T22:10:07.624 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:10:07.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:10:07.625 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-06-27T22:10:07.626 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:10:07.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836487 2024-06-27T22:10:07.627 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836487 2024-06-27T22:10:07.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-06-27T22:10:07.627 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:10:08.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836487 2024-06-27T22:10:08.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:10:09.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:10:09.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:10:09.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836487 2024-06-27T22:10:09.574 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:10:09.574 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:10:09.575 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:10:10.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:10:10.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:10:10.150 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:10:10.150 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:10:10.150 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:10:10.150 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:10:10.151 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:10:10.151 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:10:10.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:10:10.591 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:10:10.591 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:10:10.591 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:10:11.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:10:11.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:10:11.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:10:11.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1913: test_pg_scrub: pg_scrub 1.0 2024-06-27T22:10:11.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: local pgid=1.0 2024-06-27T22:10:11.153 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: get_last_scrub_stamp 1.0 2024-06-27T22:10:11.153 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:10:11.154 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:10:11.154 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:10:11.154 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:10:11.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local last_scrub=2024-06-27T22:10:04.532326+0000 2024-06-27T22:10:11.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: ceph pg scrub 1.0 2024-06-27T22:10:12.042 INFO:tasks.workunit.client.0.smithi042.stderr:instructing pg 1.0 on osd.0 to scrub 2024-06-27T22:10:12.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: wait_for_scrub 1.0 2024-06-27T22:10:04.532326+0000 2024-06-27T22:10:12.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-27T22:10:12.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-27T22:10:04.532326+0000 2024-06-27T22:10:12.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-27T22:10:12.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-27T22:10:12.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:10:12.061 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:10:12.061 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:10:12.061 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:10:12.061 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:10:12.061 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:10:12.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:10:04.532326+0000 '>' 2024-06-27T22:10:04.532326+0000 2024-06-27T22:10:12.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:10:13.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:10:13.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:10:13.472 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:10:13.472 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:10:13.472 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:10:13.473 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:10:13.473 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:10:13.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:10:04.532326+0000 '>' 2024-06-27T22:10:04.532326+0000 2024-06-27T22:10:13.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:10:14.910 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:10:14.910 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:10:14.911 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:10:14.911 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:10:14.911 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:10:14.911 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:10:14.912 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:10:15.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:10:12.370477+0000 '>' 2024-06-27T22:10:04.532326+0000 2024-06-27T22:10:15.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-27T22:10:15.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1914: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2024-06-27T22:10:15.340 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:10:15.340 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:10:15.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:10:15.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:10:15.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:10:15.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:10:15.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1915: test_pg_scrub: TIMEOUT=1 2024-06-27T22:10:15.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1915: test_pg_scrub: pg_scrub 1.0 2024-06-27T22:10:15.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: local pgid=1.0 2024-06-27T22:10:15.447 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: get_last_scrub_stamp 1.0 2024-06-27T22:10:15.448 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:10:15.448 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:10:15.448 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:10:15.448 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:10:15.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local last_scrub=2024-06-27T22:10:12.370477+0000 2024-06-27T22:10:15.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: ceph pg scrub 1.0 2024-06-27T22:10:16.290 INFO:tasks.workunit.client.0.smithi042.stderr:instructing pg 1.0 on osd.0 to scrub 2024-06-27T22:10:16.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: wait_for_scrub 1.0 2024-06-27T22:10:12.370477+0000 2024-06-27T22:10:16.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-27T22:10:16.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-27T22:10:12.370477+0000 2024-06-27T22:10:16.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-27T22:10:16.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-27T22:10:16.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-06-27T22:10:16.308 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:10:16.308 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:10:16.308 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:10:16.308 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:10:16.308 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:10:16.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:10:12.370477+0000 '>' 2024-06-27T22:10:12.370477+0000 2024-06-27T22:10:16.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:10:17.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:10:17.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-06-27T22:10:17.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: wait_for_scrub: return 1 2024-06-27T22:10:17.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1916: test_pg_scrub: teardown td/ceph-helpers 2024-06-27T22:10:17.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:10:17.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:10:17.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:10:17.728 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:10:17.728 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:10:17.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:10:17.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:10:17.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:10:17.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:10:17.838 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:10:17.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:10:17.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:10:17.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:10:17.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:10:17.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:10:17.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:10:17.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:10:17.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:10:17.842 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:10:17.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:10:17.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:10:17.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:10:17.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:10:17.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:10:17.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:10:17.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:10:17.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:10:17.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:10:17.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:10:17.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_repair td/ceph-helpers 2024-06-27T22:10:17.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1862: test_repair: local dir=td/ceph-helpers 2024-06-27T22:10:17.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1864: test_repair: setup td/ceph-helpers 2024-06-27T22:10:17.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:10:17.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:10:17.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:10:17.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:10:17.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:10:17.854 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:10:17.854 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:10:17.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:10:17.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:10:17.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:10:17.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:10:17.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:10:17.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:10:17.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:10:17.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:10:17.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:10:17.858 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:10:17.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:10:17.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:10:17.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:10:17.859 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:10:17.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:10:17.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:10:17.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:10:17.862 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:10:17.863 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:10:17.863 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:10:17.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:10:17.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:10:17.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:10:17.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:10:17.865 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:10:17.865 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:10:17.865 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:10:17.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:10:17.866 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:10:17.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:10:17.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:10:17.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:10:17.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1865: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:10:17.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:10:17.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:10:17.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:10:17.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:10:17.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:10:17.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:10:18.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:10:18.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:10:18.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:10:18.349 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:10:18.349 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:10:18.349 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:10:18.349 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:10:18.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:10:18.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:10:18.579 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:10:18.579 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:10:18.579 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:10:18.579 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:10:18.579 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:10:18.580 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:10:18.580 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:10:18.580 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:10:18.580 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:10:18.580 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:10:18.580 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:10:18.580 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:10:18.581 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:10:18.581 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:10:18.698 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:10:18.699 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:10:18.699 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:10:18.699 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:10:18.700 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:10:18.700 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:10:18.700 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:10:18.700 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:10:18.700 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:10:18.700 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:10:18.700 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:10:18.700 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:10:18.701 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:10:18.701 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:10:18.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1866: test_repair: run_mgr td/ceph-helpers x 2024-06-27T22:10:18.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:10:18.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:10:18.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:10:18.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:10:18.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:10:18.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:10:19.735 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:10:19.735 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:10:19.735 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:10:19.735 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:10:19.736 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:10:19.736 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:10:19.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:10:19.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:10:19.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:10:19.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1867: test_repair: run_osd td/ceph-helpers 0 2024-06-27T22:10:19.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:10:19.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:10:19.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:10:19.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:10:19.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:10:19.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:10:19.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:10:19.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:10:19.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:10:19.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:10:19.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:10:19.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:10:19.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:10:19.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:10:19.818 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:10:19.818 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:10:19.818 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:10:19.819 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:10:19.819 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:10:19.819 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:10:19.819 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:10:19.820 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:10:19.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d8732e79-a4b4-4c0d-9691-00d1ca0bf42b 2024-06-27T22:10:19.821 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 d8732e79-a4b4-4c0d-9691-00d1ca0bf42b 2024-06-27T22:10:19.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d8732e79-a4b4-4c0d-9691-00d1ca0bf42b' 2024-06-27T22:10:19.822 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:10:19.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBL431mOp8VMhAAvAywUDrwJj4T9BsUXE3CzQ== 2024-06-27T22:10:19.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBL431mOp8VMhAAvAywUDrwJj4T9BsUXE3CzQ=="}' 2024-06-27T22:10:19.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d8732e79-a4b4-4c0d-9691-00d1ca0bf42b -i td/ceph-helpers/0/new.json 2024-06-27T22:10:20.155 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:10:20.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:10:20.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBL431mOp8VMhAAvAywUDrwJj4T9BsUXE3CzQ== --osd-uuid d8732e79-a4b4-4c0d-9691-00d1ca0bf42b 2024-06-27T22:10:20.202 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:20.199+0000 7f19062787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:10:20.232 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:20.231+0000 7f19062787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:10:20.233 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:20.231+0000 7f19062787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:10:20.234 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:20.231+0000 7f19062787c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:10:20.234 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:20.231+0000 7f19062787c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:10:20.234 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:20.231+0000 7f19062787c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:10:20.263 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:20.263+0000 7f19062787c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:10:26.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:10:26.838 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:10:26.840 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:10:26.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:10:26.840 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:10:27.456 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:10:27.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:10:27.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:10:27.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:10:27.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:10:27.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:10:27.490 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:27.487+0000 7efdd18937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:10:27.506 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:27.503+0000 7efdd18937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:10:27.508 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:27.507+0000 7efdd18937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:10:27.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:10:27.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:10:27.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:10:27.900 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:10:27.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:10:27.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:10:27.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:27.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:10:27.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:27.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:28.072 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:28.071+0000 7efdd18937c0 -1 Falling back to public interface 2024-06-27T22:10:28.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:29.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:29.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:29.336 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:10:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:10:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:29.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:29.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:30.654 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:10:30.651+0000 7efdd18937c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:10:30.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:30.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:30.807 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:10:30.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:10:30.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:30.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:31.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:32.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:32.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:32.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:10:32.396 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:10:32.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:32.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:32.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:33.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:33.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:33.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:10:33.849 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:10:33.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:33.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:34.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:35.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:35.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:35.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:10:35.352 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:10:35.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:35.352 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:35.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:36.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:36.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:36.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:10:36.864 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:10:36.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:36.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:37.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:38.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:38.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:38.327 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:10:38.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:10:38.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:38.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:38.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:39.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:39.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:39.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:10:39.844 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:10:39.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:39.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:40.292 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:41.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:41.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:41.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:10:41.295 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:10:41.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:41.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:41.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:42.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:42.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:42.863 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:10:42.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:10:42.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:42.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:43.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:44.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:44.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:44.306 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:10:44.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:10:44.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:45.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:46.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:46.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:46.055 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:10:46.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:10:46.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:46.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:46.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:47.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:47.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:47.784 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:10:47.784 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:10:47.784 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:47.784 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:48.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:49.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:49.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:49.534 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:10:49.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:10:49.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:49.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:50.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:51.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:51.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:51.673 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:10:51.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:10:51.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:51.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:52.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:53.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:53.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:53.557 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:10:53.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:10:53.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:53.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:54.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:10:55.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:10:55.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:10:55.444 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:10:55.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:10:55.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:10:55.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:10:55.944 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1076550223,v1:127.0.0.1:6803/1076550223] [v2:127.0.0.1:6804/1076550223,v1:127.0.0.1:6805/1076550223] exists,up d8732e79-a4b4-4c0d-9691-00d1ca0bf42b 2024-06-27T22:10:55.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:10:55.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:10:55.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:10:55.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1868: test_repair: create_rbd_pool 2024-06-27T22:10:55.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:10:56.403 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:10:56.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:10:56.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:10:57.034 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:10:57.052 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:10:58.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:10:59.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1869: test_repair: wait_for_clean 2024-06-27T22:10:59.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:10:59.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:10:59.323 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:10:59.323 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:10:59.324 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:10:59.324 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:10:59.324 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:10:59.324 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:10:59.324 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:10:59.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:10:59.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:10:59.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:10:59.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:10:59.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:10:59.450 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:10:59.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:10:59.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:10:59.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:10:59.924 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:11:00.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836486 2024-06-27T22:11:00.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836486 2024-06-27T22:11:00.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836486' 2024-06-27T22:11:00.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:11:00.126 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836486 2024-06-27T22:11:00.126 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:11:00.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:11:00.128 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836486 2024-06-27T22:11:00.128 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:11:00.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836486 2024-06-27T22:11:00.129 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836486 2024-06-27T22:11:00.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-06-27T22:11:00.129 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:11:00.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836484 -lt 21474836486 2024-06-27T22:11:00.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:11:01.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:11:01.547 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:11:01.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836486 -lt 21474836486 2024-06-27T22:11:01.983 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:11:01.983 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:11:01.983 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:11:02.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:11:02.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:11:02.572 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:11:02.572 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:11:02.573 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:11:02.573 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:11:02.573 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:11:02.573 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:11:02.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:11:03.000 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:11:03.000 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:11:03.000 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:11:03.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:11:03.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:11:03.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:11:03.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1870: test_repair: repair 1.0 2024-06-27T22:11:03.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-06-27T22:11:03.547 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-06-27T22:11:03.547 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:11:03.547 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:11:03.548 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:11:03.548 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:11:03.979 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-27T22:10:56.828842+0000 2024-06-27T22:11:03.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-06-27T22:11:04.384 INFO:tasks.workunit.client.0.smithi042.stderr:instructing pg 1.0 on osd.0 to repair 2024-06-27T22:11:04.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-06-27T22:10:56.828842+0000 2024-06-27T22:11:04.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-27T22:11:04.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-27T22:10:56.828842+0000 2024-06-27T22:11:04.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-27T22:11:04.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-27T22:11:04.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:11:04.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:11:04.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:11:04.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:11:04.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:11:04.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:11:04.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:10:56.828842+0000 '>' 2024-06-27T22:10:56.828842+0000 2024-06-27T22:11:04.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:11:05.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:11:05.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:11:05.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:11:05.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:11:05.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:11:05.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:11:05.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:11:06.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:10:56.828842+0000 '>' 2024-06-27T22:10:56.828842+0000 2024-06-27T22:11:06.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:11:07.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:11:07.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:11:07.255 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:11:07.255 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:11:07.255 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:11:07.255 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:11:07.255 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:11:07.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:11:04.710803+0000 '>' 2024-06-27T22:10:56.828842+0000 2024-06-27T22:11:07.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-27T22:11:07.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1871: test_repair: kill_daemons td/ceph-helpers KILL osd 2024-06-27T22:11:07.694 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:11:07.694 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:11:07.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:11:07.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:11:07.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:11:07.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:11:07.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_repair: TIMEOUT=1 2024-06-27T22:11:07.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_repair: repair 1.0 2024-06-27T22:11:07.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-06-27T22:11:07.802 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-06-27T22:11:07.802 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:11:07.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:11:07.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:11:07.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:11:08.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-06-27T22:11:04.710803+0000 2024-06-27T22:11:08.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-06-27T22:11:08.605 INFO:tasks.workunit.client.0.smithi042.stderr:instructing pg 1.0 on osd.0 to repair 2024-06-27T22:11:08.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-06-27T22:11:04.710803+0000 2024-06-27T22:11:08.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-27T22:11:08.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-27T22:11:04.710803+0000 2024-06-27T22:11:08.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-27T22:11:08.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-27T22:11:08.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-06-27T22:11:08.622 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:11:08.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:11:08.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:11:08.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:11:08.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:11:09.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:11:04.710803+0000 '>' 2024-06-27T22:11:04.710803+0000 2024-06-27T22:11:09.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:11:10.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:11:10.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-06-27T22:11:10.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: wait_for_scrub: return 1 2024-06-27T22:11:10.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_repair: teardown td/ceph-helpers 2024-06-27T22:11:10.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:11:10.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:11:10.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:11:10.061 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:11:10.061 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:11:10.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:11:10.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:11:10.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:11:10.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:11:10.171 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:11:10.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:11:10.172 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:11:10.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:11:10.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:11:10.173 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:11:10.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:11:10.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:11:10.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:11:10.174 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:11:10.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:11:10.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:11:10.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:11:10.183 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:11:10.183 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:11:10.183 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:11:10.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:11:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:11:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:11:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:11:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_run_in_background td/ceph-helpers 2024-06-27T22:11:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: test_run_in_background: local pids 2024-06-27T22:11:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: test_run_in_background: run_in_background pids sleep 1 2024-06-27T22:11:10.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 130339"' 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 130339' 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: test_run_in_background: run_in_background pids sleep 1 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sleep 1 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 130341"' 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 130341' 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/130342: /' 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sleep 1 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/130345: /' 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_run_in_background: echo 130339 130341 2024-06-27T22:11:10.187 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_run_in_background: wc -w 2024-06-27T22:11:10.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_run_in_background: test 2 = 2 2024-06-27T22:11:10.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: test_run_in_background: wait 130339 130341 2024-06-27T22:11:11.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-27T22:11:11.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-27T22:11:11.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:11:11.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_run_mon td/ceph-helpers 2024-06-27T22:11:11.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:495: test_run_mon: local dir=td/ceph-helpers 2024-06-27T22:11:11.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:497: test_run_mon: setup td/ceph-helpers 2024-06-27T22:11:11.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:11:11.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:11:11.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:11:11.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:11:11.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:11:11.189 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:11:11.189 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:11:11.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:11:11.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:11:11.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:11:11.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:11:11.192 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:11:11.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:11:11.193 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:11:11.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:11:11.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:11:11.195 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:11:11.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:11:11.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:11:11.197 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:11:11.197 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:11:11.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:11:11.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:11:11.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:11:11.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:11:11.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:11:11.202 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:11:11.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:11:11.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:11:11.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:11:11.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:11:11.205 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:11:11.205 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:11:11.205 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:11:11.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:11:11.209 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:11:11.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:11:11.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:11:11.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:11:11.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: run_mon td/ceph-helpers a 2024-06-27T22:11:11.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:11:11.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:11:11.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:11:11.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:11:11.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:11:11.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-27T22:11:11.709 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:11:11.709 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:11:11.709 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:11:11.710 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:11:11.710 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:11:11.710 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:11:11.710 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:11:11.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T22:11:11.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:11:11.919 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:11:11.919 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:11:11.919 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:11:11.920 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:11:11.920 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:11:11.920 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:11:11.920 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:11:11.920 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:11:11.921 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:11:11.921 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:11:11.921 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:11:11.921 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:11:11.921 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:11:11.922 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:11:12.039 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:11:12.039 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:11:12.039 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:11:12.039 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:11:12.039 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:11:12.040 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:11:12.040 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:11:12.040 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:11:12.040 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:11:12.040 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:11:12.040 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:11:12.041 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:11:12.041 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:11:12.041 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:11:12.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: ceph mon dump 2024-06-27T22:11:12.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: grep mon.a 2024-06-27T22:11:12.985 INFO:tasks.workunit.client.0.smithi042.stderr:dumped monmap epoch 1 2024-06-27T22:11:13.001 INFO:tasks.workunit.client.0.smithi042.stdout:0: v2:127.0.0.1:7109/0 mon.a 2024-06-27T22:11:13.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: kill_daemons td/ceph-helpers 2024-06-27T22:11:13.002 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:11:13.002 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:11:13.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:11:13.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:11:13.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:11:13.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:11:13.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:503: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=3 2024-06-27T22:11:13.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:11:13.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:11:13.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:11:13.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:11:13.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:11:13.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=3 2024-06-27T22:11:13.344 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:13.342+0000 7fd9982d7c40 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-06-27T22:11:13.347 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:11:13.347 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:11:13.347 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:11:13.348 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:11:13.348 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:11:13.348 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:11:13.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:11:13.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-06-27T22:11:13.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:11:13.580 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:11:13.580 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:11:13.580 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:11:13.580 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:11:13.581 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:11:13.581 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:11:13.581 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:11:13.581 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:11:13.582 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:11:13.582 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:11:13.582 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:11:13.582 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:11:13.582 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:11:13.582 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:11:13.700 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:11:13.700 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:11:13.700 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:11:13.700 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:11:13.701 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:11:13.701 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:11:13.701 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:11:13.701 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:11:13.701 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:11:13.701 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:11:13.701 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:11:13.701 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:11:13.702 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:11:13.702 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:11:13.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: run_osd td/ceph-helpers 0 2024-06-27T22:11:13.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:11:13.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:11:13.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:11:13.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:11:13.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:11:13.823 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:11:13.823 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:11:13.823 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:11:13.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:11:13.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:11:13.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:11:13.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:11:13.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:11:13.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:11:13.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:11:13.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:11:13.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:11:13.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:11:13.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:11:13.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:11:13.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:11:13.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:11:13.825 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:11:13.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=532c7838-ab60-4f75-af57-def56827e2bc 2024-06-27T22:11:13.825 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 532c7838-ab60-4f75-af57-def56827e2bc 2024-06-27T22:11:13.826 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 532c7838-ab60-4f75-af57-def56827e2bc' 2024-06-27T22:11:13.826 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:11:13.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCB431mCo2cMhAA6fcv8A6vOLS24kjPI890ew== 2024-06-27T22:11:13.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCB431mCo2cMhAA6fcv8A6vOLS24kjPI890ew=="}' 2024-06-27T22:11:13.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 532c7838-ab60-4f75-af57-def56827e2bc -i td/ceph-helpers/0/new.json 2024-06-27T22:11:14.160 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:11:14.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:11:14.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCB431mCo2cMhAA6fcv8A6vOLS24kjPI890ew== --osd-uuid 532c7838-ab60-4f75-af57-def56827e2bc 2024-06-27T22:11:14.214 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:14.210+0000 7f47300f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:14.229 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:14.226+0000 7f47300f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:14.230 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:14.226+0000 7f47300f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:14.230 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:14.226+0000 7f47300f87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:11:14.231 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:14.226+0000 7f47300f87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:11:14.231 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:14.226+0000 7f47300f87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:11:14.269 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:14.266+0000 7f47300f87c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:11:20.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:11:20.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:11:20.208 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:11:20.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:11:20.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:11:20.463 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:11:20.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:11:20.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:11:20.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:11:20.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:11:20.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:11:20.496 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:20.494+0000 7fd1fa3ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:20.499 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:20.494+0000 7fd1fa3ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:20.500 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:20.498+0000 7fd1fa3ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:20.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:11:20.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:11:20.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:11:20.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:11:20.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:11:20.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:20.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:11:20.704 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:11:20.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:20.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:20.939 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:21.054 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:21.050+0000 7fd1fa3ab7c0 -1 Falling back to public interface 2024-06-27T22:11:21.941 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:21.941 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:21.941 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:11:21.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:11:21.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:21.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:22.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:23.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:23.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:23.185 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:11:23.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:11:23.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:23.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:23.427 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:23.813 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:23.810+0000 7fd1fa3ab7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:11:24.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:24.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:24.430 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:11:24.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:11:24.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:24.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:24.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:25.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:25.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:25.704 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:11:25.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:11:25.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:25.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:25.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:26.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:26.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:26.953 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:11:26.953 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:11:26.953 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:26.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:27.197 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:28.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:28.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:28.199 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:11:28.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:11:28.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:28.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:28.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:29.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:29.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:29.446 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:11:29.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:11:29.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:29.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:29.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:30.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:30.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:30.689 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:11:30.689 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:11:30.689 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:30.689 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:30.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:31.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:31.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:31.935 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:11:31.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:11:31.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:31.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:32.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:33.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:33.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:33.178 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:11:33.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:11:33.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:33.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:33.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:34.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:34.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:34.421 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:11:34.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:11:34.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:34.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:34.667 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:35.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:35.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:35.669 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:11:35.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:11:35.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:35.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:35.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:36.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:36.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:36.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:11:36.914 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:11:36.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:36.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:37.158 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:38.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:38.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:38.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:11:38.160 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:11:38.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:38.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:38.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:39.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:39.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:39.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:11:39.408 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:11:39.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:39.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:39.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:40.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:40.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:40.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:11:40.656 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:11:40.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:40.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:40.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:41.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:41.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:41.905 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:11:41.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:11:41.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:41.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:42.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:43.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:43.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:43.152 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:11:43.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:11:43.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:43.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:43.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:44.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:44.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:44.398 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T22:11:44.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T22:11:44.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:44.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:44.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:45.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:45.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:45.642 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T22:11:45.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T22:11:45.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:45.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:45.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:46.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:46.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:46.888 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T22:11:46.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T22:11:46.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:46.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:11:47.132 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3951200110,v1:127.0.0.1:6801/3951200110] [v2:127.0.0.1:6802/3951200110,v1:127.0.0.1:6803/3951200110] exists,up 532c7838-ab60-4f75-af57-def56827e2bc 2024-06-27T22:11:47.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:11:47.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:11:47.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:11:47.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: run_osd td/ceph-helpers 1 2024-06-27T22:11:47.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:11:47.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:11:47.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:11:47.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:11:47.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-27T22:11:47.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:11:47.136 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:11:47.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:11:47.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:11:47.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:11:47.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-27T22:11:47.138 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:11:47.138 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 b4437c25-d920-4454-b3dc-842e686d3b7a 2024-06-27T22:11:47.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4437c25-d920-4454-b3dc-842e686d3b7a 2024-06-27T22:11:47.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b4437c25-d920-4454-b3dc-842e686d3b7a' 2024-06-27T22:11:47.139 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:11:47.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCj431mH+2eCRAAcEhbNvFmVnMaTzRtXq2vjg== 2024-06-27T22:11:47.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCj431mH+2eCRAAcEhbNvFmVnMaTzRtXq2vjg=="}' 2024-06-27T22:11:47.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4437c25-d920-4454-b3dc-842e686d3b7a -i td/ceph-helpers/1/new.json 2024-06-27T22:11:47.530 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:11:47.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-27T22:11:47.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCj431mH+2eCRAAcEhbNvFmVnMaTzRtXq2vjg== --osd-uuid b4437c25-d920-4454-b3dc-842e686d3b7a 2024-06-27T22:11:47.585 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:47.582+0000 7f161fb577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:47.623 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:47.622+0000 7f161fb577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:47.625 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:47.622+0000 7f161fb577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:47.625 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:47.622+0000 7f161fb577c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:11:47.626 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:47.622+0000 7f161fb577c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:11:47.626 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:47.622+0000 7f161fb577c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:11:47.663 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:47.658+0000 7f161fb577c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-27T22:11:54.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-27T22:11:54.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:11:54.015 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:11:54.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:11:54.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:11:54.268 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:11:54.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:11:54.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:11:54.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:11:54.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:11:54.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:11:54.303 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:54.298+0000 7fd3ee6b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:54.305 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:54.302+0000 7fd3ee6b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:54.306 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:54.302+0000 7fd3ee6b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:11:54.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:11:54.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:11:54.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:11:54.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:11:54.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:11:54.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:54.529 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:11:54.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:11:54.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:54.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:11:54.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:54.856 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:54.854+0000 7fd3ee6b57c0 -1 Falling back to public interface 2024-06-27T22:11:55.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:55.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:55.799 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:11:55.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:11:55.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:55.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:11:56.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:57.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:57.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:57.040 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:11:57.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:11:57.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:57.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:11:57.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:57.439 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:11:57.434+0000 7fd3ee6b57c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:11:58.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:58.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:58.288 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:11:58.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:11:58.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:58.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:11:58.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:11:59.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:11:59.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:11:59.647 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:11:59.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:11:59.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:11:59.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:11:59.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:00.892 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:00.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:00.893 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:12:00.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:12:00.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:00.893 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:01.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:02.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:02.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:02.137 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:12:02.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:12:02.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:02.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:02.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:03.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:03.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:03.381 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:12:03.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:12:03.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:03.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:03.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:04.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:04.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:04.626 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:12:04.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:12:04.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:04.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:04.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:05.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:05.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:05.873 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:12:05.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:12:05.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:05.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:06.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:07.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:07.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:07.120 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:12:07.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:12:07.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:07.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:07.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:08.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:08.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:08.365 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:12:08.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:12:08.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:08.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:08.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:09.614 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:09.614 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:09.615 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:12:09.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:12:09.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:09.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:09.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:10.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:10.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:10.860 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:12:10.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:12:10.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:10.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:11.106 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:12.108 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:12.108 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:12.108 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:12:12.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:12:12.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:12.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:12.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:13.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:13.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:13.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:12:13.403 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:12:13.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:13.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:13.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:14.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:14.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:14.601 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:12:14.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:12:14.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:14.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:14.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:15.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:15.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:15.847 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:12:15.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:12:15.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:15.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:16.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:17.089 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:17.089 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:17.090 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:12:17.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:12:17.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:17.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:17.331 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:18.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:18.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:18.334 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T22:12:18.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T22:12:18.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:18.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:18.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:19.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:19.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:19.576 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T22:12:19.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T22:12:19.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:19.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:19.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:20.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:20.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:20.823 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T22:12:20.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T22:12:20.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:20.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:12:21.065 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/230235742,v1:127.0.0.1:6809/230235742] [v2:127.0.0.1:6810/230235742,v1:127.0.0.1:6811/230235742] exists,up b4437c25-d920-4454-b3dc-842e686d3b7a 2024-06-27T22:12:21.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:12:21.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:12:21.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:12:21.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:506: test_run_mon: run_osd td/ceph-helpers 2 2024-06-27T22:12:21.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:12:21.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:12:21.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-27T22:12:21.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:12:21.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-06-27T22:12:21.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:12:21.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:12:21.069 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:12:21.070 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:12:21.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:12:21.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:12:21.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:12:21.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:12:21.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:12:21.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:12:21.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:12:21.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:12:21.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:12:21.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:12:21.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:12:21.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:12:21.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-06-27T22:12:21.071 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:12:21.072 INFO:tasks.workunit.client.0.smithi042.stdout:add osd2 584c3cfe-1f68-4a9b-9c53-89799cbd08a4 2024-06-27T22:12:21.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=584c3cfe-1f68-4a9b-9c53-89799cbd08a4 2024-06-27T22:12:21.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 584c3cfe-1f68-4a9b-9c53-89799cbd08a4' 2024-06-27T22:12:21.072 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:12:21.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDF431mK4SyBRAApq1wc50yCrjpwrIZsx/V3Q== 2024-06-27T22:12:21.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDF431mK4SyBRAApq1wc50yCrjpwrIZsx/V3Q=="}' 2024-06-27T22:12:21.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 584c3cfe-1f68-4a9b-9c53-89799cbd08a4 -i td/ceph-helpers/2/new.json 2024-06-27T22:12:21.428 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:12:21.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-06-27T22:12:21.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDF431mK4SyBRAApq1wc50yCrjpwrIZsx/V3Q== --osd-uuid 584c3cfe-1f68-4a9b-9c53-89799cbd08a4 2024-06-27T22:12:21.485 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:21.482+0000 7f42ffa407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:12:21.538 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:21.534+0000 7f42ffa407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:12:21.540 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:21.538+0000 7f42ffa407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:12:21.540 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:21.538+0000 7f42ffa407c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-27T22:12:21.540 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:21.538+0000 7f42ffa407c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-27T22:12:21.540 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:21.538+0000 7f42ffa407c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-27T22:12:21.644 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:21.642+0000 7f42ffa407c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-06-27T22:12:28.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-06-27T22:12:28.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:12:28.079 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd2 key to auth repository 2024-06-27T22:12:28.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-27T22:12:28.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:12:28.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-27T22:12:28.333 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T22:12:28.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:12:28.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:12:28.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:12:28.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:12:28.367 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:28.366+0000 7fa49da277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:12:28.370 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:28.366+0000 7fa49da277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:12:28.371 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:28.370+0000 7fa49da277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:12:28.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-27T22:12:28.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:12:28.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T22:12:28.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:12:28.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:12:28.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:28.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:12:28.573 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:12:28.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:28.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:28.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:28.918 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:28.914+0000 7fa49da277c0 -1 Falling back to public interface 2024-06-27T22:12:29.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:29.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:29.805 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:12:29.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:12:29.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:29.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:30.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:31.048 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:31.048 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:31.049 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:12:31.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:12:31.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:31.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:31.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:31.618 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:31.614+0000 7fa49da277c0 -1 osd.2 0 log_to_monitors true 2024-06-27T22:12:32.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:32.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:32.297 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:12:32.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:12:32.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:32.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:32.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:33.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:33.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:33.633 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:12:33.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:12:33.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:33.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:33.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:34.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:34.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:34.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:12:34.878 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:12:34.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:34.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:35.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:36.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:36.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:36.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:12:36.126 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:12:36.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:36.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:36.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:37.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:37.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:37.375 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:12:37.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:12:37.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:37.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:37.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:38.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:38.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:38.620 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:12:38.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:12:38.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:38.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:38.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:39.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:39.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:39.866 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:12:39.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:12:39.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:39.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:40.112 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:41.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:41.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:41.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:12:41.114 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:12:41.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:41.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:41.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:42.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:42.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:42.363 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:12:42.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:12:42.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:42.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:42.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:43.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:43.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:43.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:12:43.608 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:12:43.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:43.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:43.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:44.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:44.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:44.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:12:44.853 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:12:44.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:44.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:45.094 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:46.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:46.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:46.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:12:46.096 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:12:46.097 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:46.097 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:46.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:47.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:47.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:47.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:12:47.343 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:12:47.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:47.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:47.587 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:48.589 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:48.589 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:48.589 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:12:48.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:12:48.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:48.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:48.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:49.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:49.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:49.834 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:12:49.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:12:49.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:49.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:50.075 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:51.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:51.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:51.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:12:51.077 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:12:51.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:51.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:51.321 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:52.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:52.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:52.324 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T22:12:52.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T22:12:52.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:52.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:52.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:53.570 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:53.570 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:53.570 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T22:12:53.571 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T22:12:53.571 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:53.571 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:53.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:12:54.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:12:54.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:12:54.816 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T22:12:54.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T22:12:54.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:12:54.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:12:55.091 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/921175789,v1:127.0.0.1:6817/921175789] [v2:127.0.0.1:6818/921175789,v1:127.0.0.1:6819/921175789] exists,up 584c3cfe-1f68-4a9b-9c53-89799cbd08a4 2024-06-27T22:12:55.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:12:55.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:12:55.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:12:55.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:507: test_run_mon: create_rbd_pool 2024-06-27T22:12:55.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:12:55.319 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:12:55.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:12:55.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:12:55.703 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:12:55.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:12:56.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:12:58.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: grep 'pool 1 '\''rbd'\''' 2024-06-27T22:12:58.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: ceph osd dump 2024-06-27T22:12:59.211 INFO:tasks.workunit.client.0.smithi042.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 23 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.50 2024-06-27T22:12:59.213 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: get_asok_path mon.a 2024-06-27T22:12:59.213 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:12:59.213 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:12:59.213 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:12:59.213 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:12:59.213 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:12:59.214 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:12:59.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: CEPH_ARGS= 2024-06-27T22:12:59.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get osd_pool_default_size 2024-06-27T22:12:59.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: local 'size={"osd_pool_default_size":"3"}' 2024-06-27T22:12:59.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: test '{"osd_pool_default_size":"3"}' = '{"osd_pool_default_size":"3"}' 2024-06-27T22:12:59.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: CEPH_ARGS= 2024-06-27T22:12:59.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: ceph status 2024-06-27T22:12:59.427 INFO:tasks.workunit.client.0.smithi042.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2024-06-27T22:12:59.427 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.422+0000 7f3ce7dd4640 -1 failed for service _ceph-mon._tcp 2024-06-27T22:12:59.427 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.422+0000 7f3ce7dd4640 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-06-27T22:12:59.427 INFO:tasks.workunit.client.0.smithi042.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2024-06-27T22:12:59.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: CEPH_ARGS= 2024-06-27T22:12:59.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: ceph --conf td/ceph-helpers/ceph.conf status 2024-06-27T22:12:59.536 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.534+0000 7fdf4cfd6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-06-27T22:12:59.536 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.534+0000 7fdf4cfd6640 -1 AuthRegistry(0x7fdf48069680) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-06-27T22:12:59.536 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.534+0000 7fdf4cfd6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-06-27T22:12:59.536 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.534+0000 7fdf4cfd6640 -1 AuthRegistry(0x7fdf4cfd4f50) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-06-27T22:12:59.538 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.534+0000 7fdf4cfd6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-06-27T22:12:59.538 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.534+0000 7fdf4cfd6640 -1 AuthRegistry(0x7fdf480668a0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-06-27T22:12:59.539 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.534+0000 7fdf4cfd6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-06-27T22:12:59.539 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.534+0000 7fdf4cfd6640 -1 AuthRegistry(0x7fdf48105ad0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-06-27T22:12:59.539 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.538+0000 7fdf4cfd6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-06-27T22:12:59.539 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:12:59.538+0000 7fdf4cfd6640 -1 AuthRegistry(0x7fdf480668a0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-06-27T22:12:59.640 INFO:tasks.workunit.client.0.smithi042.stdout: cluster: 2024-06-27T22:12:59.640 INFO:tasks.workunit.client.0.smithi042.stdout: id: 0e761281-cb1c-448b-bd43-8b5efce19754 2024-06-27T22:12:59.640 INFO:tasks.workunit.client.0.smithi042.stdout: health: HEALTH_OK 2024-06-27T22:12:59.640 INFO:tasks.workunit.client.0.smithi042.stdout: 2024-06-27T22:12:59.640 INFO:tasks.workunit.client.0.smithi042.stdout: services: 2024-06-27T22:12:59.641 INFO:tasks.workunit.client.0.smithi042.stdout: mon: 1 daemons, quorum a (age 105s) 2024-06-27T22:12:59.641 INFO:tasks.workunit.client.0.smithi042.stdout: mgr: no daemons active 2024-06-27T22:12:59.641 INFO:tasks.workunit.client.0.smithi042.stdout: osd: 3 osds: 3 up (since 5s), 3 in (since 38s) 2024-06-27T22:12:59.641 INFO:tasks.workunit.client.0.smithi042.stdout: 2024-06-27T22:12:59.641 INFO:tasks.workunit.client.0.smithi042.stdout: data: 2024-06-27T22:12:59.641 INFO:tasks.workunit.client.0.smithi042.stdout: pools: 0 pools, 0 pgs 2024-06-27T22:12:59.641 INFO:tasks.workunit.client.0.smithi042.stdout: objects: 0 objects, 0 B 2024-06-27T22:12:59.641 INFO:tasks.workunit.client.0.smithi042.stdout: usage: 0 B used, 0 B / 0 B avail 2024-06-27T22:12:59.641 INFO:tasks.workunit.client.0.smithi042.stdout: pgs: 2024-06-27T22:12:59.641 INFO:tasks.workunit.client.0.smithi042.stdout: 2024-06-27T22:12:59.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:516: test_run_mon: kill_daemons td/ceph-helpers 2024-06-27T22:12:59.655 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:12:59.655 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:12:59.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:12:59.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:12:59.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:13:04.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:13:04.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:13:04.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:13:04.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:13:04.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:13:04.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:13:04.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:13:04.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:13:05.003 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:04.998+0000 7fdc484ffc40 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-06-27T22:13:05.006 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:13:05.006 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:13:05.006 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:13:05.007 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:13:05.007 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:05.007 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:05.007 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:13:05.007 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:13:05.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:13:05.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:13:05.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:13:05.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:13:05.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:13:05.213 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:13:05.214 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:13:05.214 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:13:05.214 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:13:05.214 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:13:05.214 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:05.214 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:05.214 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:13:05.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:13:05.215 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:13:05.336 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:13:05.336 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:13:05.336 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:13:05.336 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:13:05.336 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:13:05.336 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:13:05.336 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:13:05.336 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:13:05.336 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:13:05.337 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:05.337 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:05.337 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:13:05.337 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:13:05.337 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:13:05.450 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: get_asok_path mon.a 2024-06-27T22:13:05.450 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:13:05.450 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:13:05.450 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:13:05.450 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:05.451 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:05.451 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:13:05.451 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: CEPH_ARGS= 2024-06-27T22:13:05.451 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get osd_pool_default_size 2024-06-27T22:13:05.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: local 'size={"osd_pool_default_size":"1"}' 2024-06-27T22:13:05.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:521: test_run_mon: test '{"osd_pool_default_size":"1"}' = '{"osd_pool_default_size":"1"}' 2024-06-27T22:13:05.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:522: test_run_mon: kill_daemons td/ceph-helpers 2024-06-27T22:13:05.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:13:05.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:13:05.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:13:05.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:13:05.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:13:05.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:13:05.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: CEPH_ARGS='--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd_pool_default_size=2' 2024-06-27T22:13:05.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: run_mon td/ceph-helpers a 2024-06-27T22:13:05.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:13:05.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:13:05.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:13:05.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:13:05.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:13:05.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-27T22:13:05.693 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:05.690+0000 7f0106f91c40 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-06-27T22:13:05.696 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:13:05.696 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:13:05.696 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:13:05.697 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:13:05.697 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:05.697 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:05.697 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:13:05.698 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T22:13:05.902 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:13:05.903 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:13:05.903 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:13:05.903 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:13:05.903 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:13:05.904 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:13:05.904 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:13:05.904 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:13:05.904 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:13:05.905 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:13:05.905 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:05.905 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:05.905 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:13:05.906 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:13:05.906 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:13:06.013 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:13:06.013 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:13:06.013 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:13:06.013 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:13:06.014 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:13:06.014 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:13:06.014 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:13:06.014 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:13:06.015 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:13:06.015 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:06.015 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:06.015 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:13:06.015 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:13:06.015 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:13:06.129 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: get_asok_path mon.a 2024-06-27T22:13:06.129 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:13:06.129 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:13:06.129 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:13:06.129 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:06.129 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:06.129 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:13:06.129 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: CEPH_ARGS= 2024-06-27T22:13:06.129 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get osd_pool_default_size 2024-06-27T22:13:06.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: local 'size={"osd_pool_default_size":"2"}' 2024-06-27T22:13:06.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: test_run_mon: test '{"osd_pool_default_size":"2"}' = '{"osd_pool_default_size":"2"}' 2024-06-27T22:13:06.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:529: test_run_mon: kill_daemons td/ceph-helpers 2024-06-27T22:13:06.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:13:06.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:13:06.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:13:06.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:13:06.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:13:06.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:13:06.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:531: test_run_mon: teardown td/ceph-helpers 2024-06-27T22:13:06.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:13:06.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:13:06.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:13:06.558 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:13:06.559 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:13:06.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:13:06.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:13:06.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:13:06.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:13:06.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:13:06.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:13:06.569 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:13:06.570 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:13:06.570 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:13:06.570 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:13:06.571 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:13:06.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:13:06.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:13:06.572 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:13:06.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:13:06.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:13:06.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:13:06.591 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:13:06.591 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:06.591 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:06.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:13:06.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:13:06.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:13:06.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:13:06.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_run_osd td/ceph-helpers 2024-06-27T22:13:06.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: local dir=td/ceph-helpers 2024-06-27T22:13:06.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:747: test_run_osd: setup td/ceph-helpers 2024-06-27T22:13:06.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:13:06.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:13:06.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:13:06.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:13:06.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:13:06.595 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:13:06.595 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:13:06.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:13:06.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:13:06.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:13:06.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:13:06.597 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:13:06.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:13:06.598 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:13:06.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:13:06.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:13:06.599 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:13:06.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:13:06.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:13:06.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:13:06.601 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:13:06.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:13:06.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:13:06.603 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:13:06.604 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:13:06.605 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:06.605 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:06.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:13:06.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:13:06.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:13:06.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:13:06.607 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:13:06.607 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:06.607 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:06.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:13:06.609 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:13:06.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:13:06.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:13:06.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:13:06.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:749: test_run_osd: run_mon td/ceph-helpers a 2024-06-27T22:13:06.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:13:06.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:13:06.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:13:06.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:13:06.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:13:06.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-06-27T22:13:07.089 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:13:07.089 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:13:07.089 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:13:07.090 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:13:07.090 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:07.090 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:07.090 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:13:07.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-27T22:13:07.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:13:07.326 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:13:07.326 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:13:07.327 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:13:07.327 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:13:07.327 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:13:07.327 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:13:07.327 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:13:07.327 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:13:07.328 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:13:07.328 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:07.328 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:07.328 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:13:07.329 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:13:07.329 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:13:07.444 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:13:07.444 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:13:07.444 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:13:07.444 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:13:07.444 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:13:07.444 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:13:07.444 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:13:07.444 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:13:07.445 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:13:07.445 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:07.446 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:07.446 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:13:07.446 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:13:07.446 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:13:07.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: run_mgr td/ceph-helpers x 2024-06-27T22:13:07.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:13:07.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:13:07.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:13:07.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:13:07.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:13:07.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:13:08.492 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:13:08.492 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:13:08.492 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:13:08.492 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:13:08.492 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:08.492 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:08.493 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:13:08.493 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:13:08.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:13:08.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:752: test_run_osd: run_osd td/ceph-helpers 0 2024-06-27T22:13:08.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:13:08.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:13:08.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:13:08.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:13:08.567 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:13:08.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:13:08.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:13:08.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:13:08.569 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:13:08.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3e17acda-b149-4d56-8283-db4dc04b85bc 2024-06-27T22:13:08.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3e17acda-b149-4d56-8283-db4dc04b85bc' 2024-06-27T22:13:08.570 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 3e17acda-b149-4d56-8283-db4dc04b85bc 2024-06-27T22:13:08.570 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:13:08.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0431m7sQPIxAAAP69k76nmrqcjqPeC/4ALQ== 2024-06-27T22:13:08.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0431m7sQPIxAAAP69k76nmrqcjqPeC/4ALQ=="}' 2024-06-27T22:13:08.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3e17acda-b149-4d56-8283-db4dc04b85bc -i td/ceph-helpers/0/new.json 2024-06-27T22:13:08.887 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:13:08.902 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:13:08.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0431m7sQPIxAAAP69k76nmrqcjqPeC/4ALQ== --osd-uuid 3e17acda-b149-4d56-8283-db4dc04b85bc 2024-06-27T22:13:08.931 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:08.930+0000 7f37cdcbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:08.955 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:08.954+0000 7f37cdcbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:08.957 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:08.954+0000 7f37cdcbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:08.957 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:08.954+0000 7f37cdcbe7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:13:08.957 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:08.954+0000 7f37cdcbe7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:13:08.957 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:08.954+0000 7f37cdcbe7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:13:08.995 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:08.994+0000 7f37cdcbe7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:13:15.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:13:15.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:13:15.519 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:13:15.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:13:15.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:13:16.133 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:13:16.133 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:13:16.133 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:13:16.133 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:13:16.134 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:13:16.134 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:13:16.165 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:16.162+0000 7fca531af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:16.196 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:16.194+0000 7fca531af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:16.197 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:16.194+0000 7fca531af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:16.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:13:16.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:13:16.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:13:16.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:13:16.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:13:16.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:16.598 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:13:16.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:13:16.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:16.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:16.768 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:16.766+0000 7fca531af7c0 -1 Falling back to public interface 2024-06-27T22:13:17.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:18.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:18.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:18.047 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:13:18.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:13:18.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:18.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:18.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:19.338 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:19.334+0000 7fca531af7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:13:19.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:19.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:19.530 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:13:19.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:13:19.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:19.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:20.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:21.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:21.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:21.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:13:21.059 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:13:21.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:21.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:21.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:22.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:22.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:22.522 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:13:22.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:13:22.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:22.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:23.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:24.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:24.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:24.002 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:13:24.002 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:13:24.002 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:24.002 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:24.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:25.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:25.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:25.529 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:13:25.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:13:25.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:25.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:25.983 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:26.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:26.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:26.985 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:13:26.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:13:26.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:26.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:27.462 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:28.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:28.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:28.465 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:13:28.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:13:28.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:28.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:28.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:29.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:29.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:29.940 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:13:29.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:13:29.941 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:29.941 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:30.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:31.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:31.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:31.521 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:13:31.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:13:31.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:31.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:32.018 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:33.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:33.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:33.020 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:13:33.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:13:33.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:33.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:33.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:34.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:34.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:34.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:13:34.811 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:13:34.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:34.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:35.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:36.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:36.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:36.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:13:36.593 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:13:36.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:36.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:37.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:38.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:38.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:38.418 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:13:38.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:13:38.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:38.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:39.258 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:40.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:40.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:40.260 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:13:40.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:13:40.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:40.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:40.839 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:41.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:41.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:41.841 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:13:41.841 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:13:41.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:41.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:43.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:44.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:44.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:44.328 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:13:44.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:13:44.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:44.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:13:44.787 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3381413816,v1:127.0.0.1:6803/3381413816] [v2:127.0.0.1:6804/3381413816,v1:127.0.0.1:6805/3381413816] exists,up 3e17acda-b149-4d56-8283-db4dc04b85bc 2024-06-27T22:13:44.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:13:44.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:13:44.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:13:44.789 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: get_asok_path osd.0 2024-06-27T22:13:44.789 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-27T22:13:44.789 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-27T22:13:44.790 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:13:44.790 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:44.790 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:44.791 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.0.asok 2024-06-27T22:13:44.791 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: CEPH_ARGS= 2024-06-27T22:13:44.791 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-osd.0.asok config get osd_max_backfills 2024-06-27T22:13:44.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-06-27T22:13:44.916 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: echo '{"osd_max_backfills":"1"}' 2024-06-27T22:13:44.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: grep --quiet osd_max_backfills 2024-06-27T22:13:44.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:757: test_run_osd: run_osd td/ceph-helpers 1 --osd-max-backfills 20 2024-06-27T22:13:44.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:13:44.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:13:44.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:13:44.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:13:44.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:13:44.923 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:13:44.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:13:44.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-max-backfills 20' 2024-06-27T22:13:44.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-27T22:13:44.925 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:13:44.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=44331224-a061-441b-a4f2-27d502e4566e 2024-06-27T22:13:44.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 44331224-a061-441b-a4f2-27d502e4566e' 2024-06-27T22:13:44.926 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 44331224-a061-441b-a4f2-27d502e4566e 2024-06-27T22:13:44.926 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:13:44.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAY5H1mPbBAOBAAzqn0yKdPF9GR6hIOAmRoKw== 2024-06-27T22:13:44.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAY5H1mPbBAOBAAzqn0yKdPF9GR6hIOAmRoKw=="}' 2024-06-27T22:13:44.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 44331224-a061-441b-a4f2-27d502e4566e -i td/ceph-helpers/1/new.json 2024-06-27T22:13:45.450 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:13:45.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-27T22:13:45.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 --mkfs --key AQAY5H1mPbBAOBAAzqn0yKdPF9GR6hIOAmRoKw== --osd-uuid 44331224-a061-441b-a4f2-27d502e4566e 2024-06-27T22:13:45.496 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:45.494+0000 7f771c9587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:45.530 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:45.526+0000 7f771c9587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:45.532 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:45.530+0000 7f771c9587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:45.533 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:45.530+0000 7f771c9587c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:13:45.533 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:45.530+0000 7f771c9587c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:13:45.533 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:45.530+0000 7f771c9587c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:13:45.568 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:45.566+0000 7f771c9587c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-27T22:13:51.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-27T22:13:51.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:13:51.973 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:13:51.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:13:51.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:13:52.567 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:13:52.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:13:52.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-06-27T22:13:52.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:13:52.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:13:52.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:13:52.601 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:52.598+0000 7f26f494e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:52.616 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:52.614+0000 7f26f494e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:52.617 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:52.614+0000 7f26f494e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:13:53.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:13:53.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:13:53.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:13:53.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:13:53.004 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:13:53.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:13:53.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:53.005 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:13:53.005 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:53.005 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:13:53.204 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:53.202+0000 7f26f494e7c0 -1 Falling back to public interface 2024-06-27T22:13:53.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:54.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:54.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:54.453 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:13:54.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:13:54.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:54.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:13:54.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:55.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:55.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:55.942 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:13:55.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:13:55.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:55.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:13:56.010 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:13:56.006+0000 7f26f494e7c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:13:56.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:57.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:57.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:57.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:13:57.595 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:13:57.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:57.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:13:58.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:13:59.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:13:59.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:13:59.145 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:13:59.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:13:59.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:13:59.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:13:59.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:00.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:00.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:00.617 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:14:00.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:14:00.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:00.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:01.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:02.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:02.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:02.087 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:14:02.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:14:02.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:02.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:02.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:03.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:03.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:03.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:14:03.567 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:14:03.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:03.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:04.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:05.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:05.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:05.149 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:14:05.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:14:05.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:05.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:05.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:06.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:06.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:06.626 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:14:06.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:14:06.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:06.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:07.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:08.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:08.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:08.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:14:08.071 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:14:08.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:08.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:08.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:09.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:09.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:09.525 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:14:09.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:14:09.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:09.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:10.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:11.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:11.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:11.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:14:11.212 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:14:11.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:11.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:11.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:12.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:12.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:12.924 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:14:12.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:14:12.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:12.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:13.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:14.780 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:14.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:14.781 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:14:14.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:14:14.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:14.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:15.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:16.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:16.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:16.627 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:14:16.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:14:16.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:16.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:17.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:18.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:18.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:18.547 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:14:18.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:14:18.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:18.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:19.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:20.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:20.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:20.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:14:20.412 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:14:20.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:20.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:21.227 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:22.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:22.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:22.229 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:14:22.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:14:22.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:22.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:14:22.702 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/538796527,v1:127.0.0.1:6811/538796527] [v2:127.0.0.1:6812/538796527,v1:127.0.0.1:6813/538796527] exists,up 44331224-a061-441b-a4f2-27d502e4566e 2024-06-27T22:14:22.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:14:22.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:14:22.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:14:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: get_asok_path osd.1 2024-06-27T22:14:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-27T22:14:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-27T22:14:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:14:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:14:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:14:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.1.asok 2024-06-27T22:14:22.704 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: CEPH_ARGS= 2024-06-27T22:14:22.705 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-osd.1.asok config get osd_max_backfills 2024-06-27T22:14:22.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-06-27T22:14:22.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:760: test_run_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-06-27T22:14:22.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: CEPH_ARGS='--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-06-27T22:14:22.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: run_osd td/ceph-helpers 2 2024-06-27T22:14:22.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:14:22.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:14:22.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-27T22:14:22.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:14:22.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-06-27T22:14:22.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-06-27T22:14:22.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:14:22.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:14:22.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:14:22.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-06-27T22:14:22.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-06-27T22:14:22.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:14:22.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:14:22.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:14:22.830 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:14:22.831 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:14:22.831 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:14:22.833 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:14:22.833 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:14:22.833 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:14:22.833 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:14:22.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:14:22.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:14:22.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:14:22.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:14:22.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:14:22.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:14:22.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:14:22.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:14:22.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:14:22.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:14:22.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:14:22.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:14:22.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-06-27T22:14:22.834 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:14:22.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ffe06e35-7b8c-4f94-9f03-c61d09da98b4 2024-06-27T22:14:22.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ffe06e35-7b8c-4f94-9f03-c61d09da98b4' 2024-06-27T22:14:22.835 INFO:tasks.workunit.client.0.smithi042.stdout:add osd2 ffe06e35-7b8c-4f94-9f03-c61d09da98b4 2024-06-27T22:14:22.835 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:14:22.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+5H1mQ/zbMhAApPzUsWdISPRsQOR6CK7BLw== 2024-06-27T22:14:22.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+5H1mQ/zbMhAApPzUsWdISPRsQOR6CK7BLw=="}' 2024-06-27T22:14:22.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ffe06e35-7b8c-4f94-9f03-c61d09da98b4 -i td/ceph-helpers/2/new.json 2024-06-27T22:14:23.489 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:14:23.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-06-27T22:14:23.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+5H1mQ/zbMhAApPzUsWdISPRsQOR6CK7BLw== --osd-uuid ffe06e35-7b8c-4f94-9f03-c61d09da98b4 2024-06-27T22:14:23.544 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:23.542+0000 7f00decbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:14:23.578 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:23.574+0000 7f00decbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:14:23.579 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:23.578+0000 7f00decbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:14:23.580 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:23.578+0000 7f00decbb7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-27T22:14:23.580 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:23.578+0000 7f00decbb7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-27T22:14:23.580 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:23.578+0000 7f00decbb7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-27T22:14:23.685 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:23.682+0000 7f00decbb7c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-06-27T22:14:30.052 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-06-27T22:14:30.052 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:14:30.053 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd2 key to auth repository 2024-06-27T22:14:30.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-27T22:14:30.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:14:30.638 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T22:14:30.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-27T22:14:30.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:14:30.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:14:30.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:14:30.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:14:30.672 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:30.670+0000 7f52224b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:14:30.709 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:30.706+0000 7f52224b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:14:30.711 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:30.710+0000 7f52224b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:14:31.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-27T22:14:31.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:14:31.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T22:14:31.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:14:31.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:14:31.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:31.132 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:14:31.133 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:14:31.133 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:31.133 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:31.303 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:31.302+0000 7f52224b57c0 -1 Falling back to public interface 2024-06-27T22:14:31.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:32.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:32.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:32.593 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:14:32.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:14:32.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:32.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:33.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:34.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:34.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:34.038 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:14:34.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:14:34.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:34.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:34.047 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:14:34.046+0000 7f52224b57c0 -1 osd.2 0 log_to_monitors true 2024-06-27T22:14:34.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:35.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:35.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:35.812 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:14:35.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:14:35.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:35.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:36.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:37.356 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:37.356 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:37.357 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:14:37.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:14:37.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:37.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:37.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:38.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:38.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:38.820 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:14:38.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:14:38.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:38.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:39.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:40.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:40.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:40.272 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:14:40.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:14:40.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:40.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:40.753 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:41.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:41.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:41.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:14:41.756 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:14:41.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:41.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:42.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:43.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:43.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:43.338 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:14:43.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:14:43.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:43.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:43.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:44.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:44.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:44.868 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:14:44.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:14:44.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:44.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:45.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:46.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:46.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:46.316 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:14:46.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:14:46.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:46.317 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:46.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:47.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:47.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:47.761 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:14:47.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:14:47.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:47.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:48.795 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:49.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:49.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:49.798 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:14:49.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:14:49.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:49.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:50.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:51.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:51.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:51.509 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:14:51.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:14:51.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:51.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:52.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:53.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:53.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:53.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:14:53.050 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:14:53.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:53.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:53.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:54.795 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:54.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:54.796 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:14:54.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:14:54.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:54.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:55.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:56.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:56.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:56.543 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:14:56.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:14:56.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:56.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:57.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:58.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:58.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:58.417 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:14:58.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:14:58.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:58.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:14:58.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:14:59.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:14:59.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:14:59.912 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:14:59.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:14:59.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:14:59.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:15:00.345 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/26867771,v1:127.0.0.1:6819/26867771] [v2:127.0.0.1:6820/26867771,v1:127.0.0.1:6821/26867771] exists,up ffe06e35-7b8c-4f94-9f03-c61d09da98b4 2024-06-27T22:15:00.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:15:00.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:15:00.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:15:00.347 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: get_asok_path osd.2 2024-06-27T22:15:00.347 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-06-27T22:15:00.347 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-06-27T22:15:00.348 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:15:00.348 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:00.348 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:00.348 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-osd.2.asok 2024-06-27T22:15:00.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: CEPH_ARGS= 2024-06-27T22:15:00.349 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.70124/ceph-osd.2.asok config get osd_max_backfills 2024-06-27T22:15:00.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: local 'backfills={"osd_max_backfills":"30"}' 2024-06-27T22:15:00.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:765: test_run_osd: test '{"osd_max_backfills":"30"}' = '{"osd_max_backfills":"30"}' 2024-06-27T22:15:00.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:767: test_run_osd: teardown td/ceph-helpers 2024-06-27T22:15:00.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:15:00.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:15:00.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:15:00.472 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:15:00.472 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:15:00.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:15:00.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:15:00.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:15:00.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:15:00.782 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:15:00.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:15:00.784 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:15:00.785 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:15:00.785 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:15:00.786 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:15:00.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:00.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:15:00.788 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:00.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:15:00.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:15:00.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:15:00.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:15:00.808 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:15:00.808 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:00.808 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:00.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:15:00.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:15:00.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:15:00.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:15:00.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_set_config td/ceph-helpers 2024-06-27T22:15:00.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: test_set_config: local dir=td/ceph-helpers 2024-06-27T22:15:00.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: test_set_config: setup td/ceph-helpers 2024-06-27T22:15:00.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:15:00.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:15:00.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:15:00.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:15:00.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:15:00.812 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:15:00.812 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:15:00.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:15:00.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:15:00.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:15:00.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:15:00.814 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:15:00.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:15:00.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:15:00.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:15:00.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:15:00.817 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:15:00.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:00.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:15:00.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:15:00.819 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:00.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:15:00.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:15:00.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:15:00.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:15:00.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:00.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:00.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:15:00.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:15:00.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:15:00.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:15:00.826 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:15:00.826 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:00.826 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:00.826 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:15:00.827 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:15:00.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:15:00.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:15:00.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:15:00.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: test_set_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:15:00.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:15:00.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:15:00.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:15:00.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:15:00.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:15:00.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:15:01.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:15:01.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:15:01.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:15:01.307 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:15:01.307 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:01.307 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:01.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:15:01.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:15:01.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:15:01.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:15:01.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:15:01.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:15:01.526 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:15:01.527 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:15:01.527 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:15:01.527 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:15:01.527 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:15:01.527 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:15:01.527 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:01.527 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:01.528 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:15:01.528 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:15:01.528 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:15:01.644 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:15:01.645 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:15:01.645 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:15:01.645 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:15:01.646 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:15:01.646 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:15:01.646 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:15:01.646 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:15:01.647 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:15:01.647 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:01.647 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:01.647 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:15:01.647 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:15:01.647 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:15:01.760 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: get_config mon a ms_crc_header 2024-06-27T22:15:01.760 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:15:01.760 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:15:01.760 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-06-27T22:15:01.761 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-06-27T22:15:01.761 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:15:01.761 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:15:01.761 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:15:01.762 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:15:01.762 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:01.762 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:01.763 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:15:01.763 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:15:01.763 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get ms_crc_header 2024-06-27T22:15:01.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: test true = true 2024-06-27T22:15:01.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: test_set_config: set_config mon a ms_crc_header false 2024-06-27T22:15:01.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-06-27T22:15:01.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-06-27T22:15:01.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-06-27T22:15:01.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=false 2024-06-27T22:15:01.874 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-27T22:15:01.874 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-06-27T22:15:01.874 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:15:01.874 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:15:01.875 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:15:01.875 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:01.875 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:01.875 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:15:01.875 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config set ms_crc_header false 2024-06-27T22:15:01.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-27T22:15:01.995 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: get_config mon a ms_crc_header 2024-06-27T22:15:01.995 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:15:01.996 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:15:01.996 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-06-27T22:15:01.997 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-06-27T22:15:01.997 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:15:01.997 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:15:01.997 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:15:01.997 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:15:01.997 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:01.997 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:01.997 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:15:01.998 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:15:01.998 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get ms_crc_header 2024-06-27T22:15:02.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: test false = false 2024-06-27T22:15:02.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1179: test_set_config: set_config mon a ms_crc_header true 2024-06-27T22:15:02.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-06-27T22:15:02.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-06-27T22:15:02.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-06-27T22:15:02.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-27T22:15:02.105 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-27T22:15:02.105 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-06-27T22:15:02.105 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:15:02.106 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:15:02.106 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:15:02.106 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:02.106 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:02.106 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:15:02.107 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config set ms_crc_header true 2024-06-27T22:15:02.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-27T22:15:02.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: get_config mon a ms_crc_header 2024-06-27T22:15:02.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:15:02.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:15:02.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-06-27T22:15:02.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-06-27T22:15:02.220 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:15:02.220 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:15:02.220 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:15:02.221 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:15:02.221 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:02.221 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:02.221 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:15:02.222 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:15:02.222 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get ms_crc_header 2024-06-27T22:15:02.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: test true = true 2024-06-27T22:15:02.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1181: test_set_config: teardown td/ceph-helpers 2024-06-27T22:15:02.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:15:02.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:15:02.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:15:02.334 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:15:02.334 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:15:02.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:15:02.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:15:02.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:15:02.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:15:02.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:15:02.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:15:02.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:15:02.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:15:02.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:15:02.444 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:15:02.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:15:02.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:15:02.446 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:15:02.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:15:02.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:15:02.451 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:15:02.451 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:02.451 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:02.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:15:02.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:15:02.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:15:02.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:15:02.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_setup td/ceph-helpers 2024-06-27T22:15:02.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:144: test_setup: local dir=td/ceph-helpers 2024-06-27T22:15:02.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:145: test_setup: setup td/ceph-helpers 2024-06-27T22:15:02.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:15:02.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:15:02.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:15:02.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:15:02.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:15:02.454 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:15:02.455 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:15:02.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:15:02.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:15:02.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:15:02.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:15:02.456 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:15:02.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:15:02.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:15:02.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:15:02.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:15:02.459 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:15:02.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:15:02.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:15:02.462 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:15:02.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:15:02.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:15:02.466 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:15:02.466 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:02.466 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:02.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:15:02.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:15:02.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:15:02.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:15:02.469 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:15:02.469 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:02.469 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:02.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:15:02.471 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:15:02.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:15:02.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:15:02.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:146: test_setup: test -d td/ceph-helpers 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:147: test_setup: setup td/ceph-helpers 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:15:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:15:02.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:15:02.475 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:15:02.476 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:15:02.477 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:15:02.478 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:15:02.478 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:15:02.478 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:15:02.479 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.479 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:15:02.480 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:15:02.480 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.481 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:15:02.482 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:15:02.482 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:15:02.484 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:15:02.484 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:02.484 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:02.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:15:02.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:15:02.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:15:02.485 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:15:02.487 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:15:02.487 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:02.487 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:02.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:15:02.490 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:15:02.490 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:148: test_setup: test -d td/ceph-helpers 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:149: test_setup: teardown td/ceph-helpers 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:15:02.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:15:02.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:15:02.493 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:15:02.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:15:02.495 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:15:02.496 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:15:02.496 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:15:02.497 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:15:02.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:15:02.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:15:02.498 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.499 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:15:02.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:15:02.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:15:02.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:15:02.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:02.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:02.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:15:02.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:15:02.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:15:02.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:15:02.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_teardown td/ceph-helpers 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:220: test_teardown: local dir=td/ceph-helpers 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221: test_teardown: setup td/ceph-helpers 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:15:02.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:15:02.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:15:02.508 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:15:02.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:15:02.510 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:15:02.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:15:02.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:15:02.511 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:15:02.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:15:02.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:15:02.513 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:15:02.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:15:02.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:15:02.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:15:02.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:02.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:02.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:15:02.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:15:02.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:15:02.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:15:02.520 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:15:02.521 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:02.521 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:02.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:15:02.522 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:15:02.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:15:02.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:15:02.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:15:02.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222: test_teardown: teardown td/ceph-helpers 2024-06-27T22:15:02.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:15:02.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:15:02.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:15:02.524 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:15:02.524 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:15:02.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:15:02.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:15:02.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:15:02.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:15:02.528 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:15:02.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:15:02.529 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:15:02.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:15:02.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:15:02.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:15:02.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:15:02.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:15:02.533 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:02.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:15:02.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:15:02.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:15:02.536 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:15:02.537 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:02.537 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:02.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:15:02.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:15:02.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:15:02.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223: test_teardown: test -d td/ceph-helpers 2024-06-27T22:15:02.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:15:02.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_background td/ceph-helpers 2024-06-27T22:15:02.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: test_wait_background: local pids= 2024-06-27T22:15:02.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2198: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 1' 2024-06-27T22:15:02.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-27T22:15:02.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-27T22:15:02.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 142160"' 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 142160' 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 142162"' 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 142162' 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: bash -c 'sleep 1; exit 1' 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: test_wait_background: wait_background pids 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 142160 142162' 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 142160 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/142163: /' 2024-06-27T22:15:02.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: bash -c 'sleep 2; exit 0' 2024-06-27T22:15:02.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/142165: /' 2024-06-27T22:15:03.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 1 2024-06-27T22:15:03.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: wait_background: return_code=1 2024-06-27T22:15:03.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-27T22:15:03.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 142162 2024-06-27T22:15:04.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-27T22:15:04.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-27T22:15:04.545 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-27T22:15:04.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 1 2024-06-27T22:15:04.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: test_wait_background: '[' 1 -ne 1 ']' 2024-06-27T22:15:04.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 0' 2024-06-27T22:15:04.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-27T22:15:04.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-27T22:15:04.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 142168"' 2024-06-27T22:15:04.545 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 142168' 2024-06-27T22:15:04.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2204: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 142169"' 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 142169' 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: test_wait_background: wait_background pids 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 142168 142169' 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 142168 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: bash -c 'sleep 1; exit 0' 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: bash -c 'sleep 2; exit 0' 2024-06-27T22:15:04.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/142171: /' 2024-06-27T22:15:04.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/142173: /' 2024-06-27T22:15:05.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-27T22:15:05.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-06-27T22:15:05.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 142169 2024-06-27T22:15:06.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-06-27T22:15:06.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-06-27T22:15:06.550 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-06-27T22:15:06.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-06-27T22:15:06.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2206: test_wait_background: '[' 0 -ne 0 ']' 2024-06-27T22:15:06.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: test_wait_background: '[' '!' -z '' ']' 2024-06-27T22:15:06.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:15:06.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_for_clean td/ceph-helpers 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: test_wait_for_clean: local dir=td/ceph-helpers 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1679: test_wait_for_clean: setup td/ceph-helpers 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:15:06.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:15:06.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:15:06.552 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:15:06.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:15:06.554 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:15:06.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:15:06.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:15:06.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:15:06.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:06.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:15:06.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:15:06.558 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:15:06.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:15:06.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:15:06.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:15:06.562 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:15:06.562 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:06.562 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:06.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:15:06.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:15:06.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:15:06.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:15:06.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:15:06.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:06.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:06.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:15:06.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:15:06.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:15:06.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:15:06.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:15:06.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1680: test_wait_for_clean: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-06-27T22:15:06.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:15:06.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:15:06.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:15:06.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:15:06.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:15:06.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-06-27T22:15:07.088 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:15:07.088 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:15:07.088 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:15:07.089 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:15:07.089 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:07.089 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:07.089 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:15:07.089 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-27T22:15:07.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:15:07.293 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:15:07.293 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:15:07.293 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:15:07.293 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:15:07.294 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:15:07.295 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:15:07.295 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:15:07.295 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:15:07.295 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:15:07.295 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:07.295 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:07.295 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:15:07.295 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:15:07.295 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:15:07.415 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:15:07.415 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:15:07.415 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:15:07.415 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:15:07.415 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:15:07.415 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:15:07.415 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:15:07.416 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:15:07.416 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:15:07.416 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:07.416 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:07.416 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:15:07.416 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:15:07.416 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:15:07.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1681: test_wait_for_clean: run_osd td/ceph-helpers 0 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:15:07.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:15:07.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:15:07.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:15:07.521 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:15:07.521 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:15:07.521 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:15:07.522 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:15:07.522 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:07.523 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:07.523 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:15:07.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:15:07.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:15:07.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:15:07.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:15:07.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:15:07.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:15:07.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:15:07.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:15:07.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:15:07.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:15:07.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:15:07.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:15:07.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:15:07.524 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:15:07.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=989af8a7-807b-4b46-a149-46eb5075cb94 2024-06-27T22:15:07.525 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 989af8a7-807b-4b46-a149-46eb5075cb94 2024-06-27T22:15:07.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 989af8a7-807b-4b46-a149-46eb5075cb94' 2024-06-27T22:15:07.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:15:07.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBr5H1mSEVYIBAA3kI3mWA3slx7ZxgzRYCCyw== 2024-06-27T22:15:07.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBr5H1mSEVYIBAA3kI3mWA3slx7ZxgzRYCCyw=="}' 2024-06-27T22:15:07.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 989af8a7-807b-4b46-a149-46eb5075cb94 -i td/ceph-helpers/0/new.json 2024-06-27T22:15:08.477 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:15:08.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:15:08.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBr5H1mSEVYIBAA3kI3mWA3slx7ZxgzRYCCyw== --osd-uuid 989af8a7-807b-4b46-a149-46eb5075cb94 2024-06-27T22:15:08.524 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:08.522+0000 7f469b74b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:15:08.545 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:08.542+0000 7f469b74b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:15:08.546 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:08.546+0000 7f469b74b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:15:08.547 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:08.546+0000 7f469b74b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:15:08.547 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:08.546+0000 7f469b74b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:15:08.547 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:08.546+0000 7f469b74b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:15:08.585 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:08.582+0000 7f469b74b7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:15:14.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:15:14.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:15:14.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:15:14.738 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:15:14.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:15:14.994 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:15:14.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:15:14.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:15:14.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:15:14.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:15:14.996 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:15:15.029 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:15.026+0000 7f024a7317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:15:15.032 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:15.030+0000 7f024a7317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:15:15.033 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:15.030+0000 7f024a7317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:15:15.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:15:15.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:15:15.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:15:15.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:15:15.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:15:15.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:15.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:15:15.229 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:15:15.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:15.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:15.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:15.584 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:15.582+0000 7f024a7317c0 -1 Falling back to public interface 2024-06-27T22:15:16.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:16.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:16.467 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:15:16.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:15:16.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:16.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:16.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:17.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:17.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:17.712 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:15:17.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:15:17.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:17.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:17.954 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:18.270 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:18.270+0000 7f024a7317c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:15:18.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:18.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:18.956 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:15:18.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:15:18.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:18.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:19.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:20.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:20.328 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:15:20.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:20.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:15:20.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:20.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:20.571 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:21.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:21.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:21.573 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:15:21.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:15:21.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:21.574 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:21.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:22.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:22.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:22.820 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:15:22.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:15:22.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:22.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:23.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:24.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:24.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:24.064 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:15:24.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:15:24.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:24.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:24.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:25.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:25.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:25.312 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:15:25.312 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:15:25.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:25.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:25.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:26.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:26.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:26.562 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:15:26.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:15:26.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:26.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:26.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:27.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:27.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:27.809 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:15:27.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:15:27.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:27.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:28.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:29.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:29.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:29.057 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:15:29.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:15:29.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:29.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:29.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:30.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:30.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:30.306 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:15:30.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:15:30.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:30.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:30.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:31.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:31.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:31.556 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:15:31.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:15:31.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:31.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:31.798 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:32.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:32.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:32.802 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:15:32.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:15:32.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:32.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:33.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:34.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:34.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:34.047 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:15:34.048 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:15:34.048 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:34.048 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:34.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:35.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:35.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:35.296 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:15:35.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:15:35.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:35.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:35.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:36.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:36.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:36.544 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:15:36.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:15:36.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:36.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:36.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:37.793 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:37.793 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:37.793 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:15:37.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:15:37.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:37.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:38.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:39.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:39.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:39.037 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T22:15:39.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T22:15:39.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:39.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:39.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:40.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:40.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:40.283 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T22:15:40.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T22:15:40.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:40.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:40.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:15:41.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:15:41.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:15:41.596 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T22:15:41.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T22:15:41.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:15:41.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:15:41.841 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3805929884,v1:127.0.0.1:6801/3805929884] [v2:127.0.0.1:6802/3805929884,v1:127.0.0.1:6803/3805929884] exists,up 989af8a7-807b-4b46-a149-46eb5075cb94 2024-06-27T22:15:41.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:15:41.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:15:41.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:15:41.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1682: test_wait_for_clean: run_mgr td/ceph-helpers x 2024-06-27T22:15:41.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:15:41.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:15:41.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:15:41.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:15:41.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:15:41.843 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:15:42.143 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:15:42.143 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:15:42.144 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:15:42.144 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:15:42.144 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:42.144 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:42.144 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:15:42.145 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:15:42.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:15:42.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: test_wait_for_clean: create_rbd_pool 2024-06-27T22:15:42.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:15:42.405 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:15:42.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:15:42.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:15:42.812 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:15:42.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:15:43.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:15:44.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: WAIT_FOR_CLEAN_TIMEOUT=1 2024-06-27T22:15:44.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: wait_for_clean 2024-06-27T22:15:44.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:15:44.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:15:44.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:15:44.943 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-06-27T22:15:44.944 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:15:44.944 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:15:44.944 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:15:44.944 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:15:44.944 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:15:44.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-06-27T22:15:44.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:15:44.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:15:44.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:15:44.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:15:44.978 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:15:45.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:15:45.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:15:45.259 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:15:45.259 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:15:45.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=0 2024-06-27T22:15:45.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 0 2024-06-27T22:15:45.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-0' 2024-06-27T22:15:45.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:15:45.406 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-0 2024-06-27T22:15:45.406 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:15:45.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:15:45.407 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-0 2024-06-27T22:15:45.407 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:15:45.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=0 2024-06-27T22:15:45.408 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 0 2024-06-27T22:15:45.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-06-27T22:15:45.408 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:15:45.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 0 -lt 0 2024-06-27T22:15:45.633 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:15:45.633 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:15:45.634 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:15:45.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-06-27T22:15:45.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-06-27T22:15:46.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:15:46.869 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:15:46.869 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:15:47.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-06-27T22:15:47.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-06-27T22:15:48.142 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:15:48.142 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:15:48.143 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:15:48.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:15:48.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:15:48.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:15:48.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:15:48.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:15:48.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:15:48.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:15:48.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:15:49.134 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-27T22:15:49.135 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:15:49.135 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:15:49.135 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:15:49.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-27T22:15:49.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-27T22:15:49.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-27T22:15:49.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-27T22:15:49.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-27T22:15:49.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-27T22:15:49.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-27T22:15:49.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:15:49.804 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:15:49.804 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:15:49.804 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:15:49.804 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:15:49.805 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:15:49.805 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:15:50.225 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-27T22:15:50.226 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:15:50.226 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:15:50.226 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:15:50.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-27T22:15:50.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-27T22:15:50.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-27T22:15:50.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-27T22:15:50.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-27T22:15:50.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-27T22:15:50.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-27T22:15:50.782 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-27T22:15:50.782 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-27T22:15:51.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-27T22:15:51.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-27T22:15:51.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-06-27T22:15:51.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-27T22:15:51.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-27T22:15:51.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-27T22:15:51.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:15:51.601 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:15:51.602 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:15:51.602 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:15:51.602 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:15:51.602 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:15:51.602 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:15:52.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-27T22:15:52.027 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:15:52.028 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:15:52.028 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:15:52.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-27T22:15:52.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-27T22:15:52.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-27T22:15:52.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-27T22:15:52.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-27T22:15:52.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-27T22:15:52.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-27T22:15:52.594 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-27T22:15:52.594 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-27T22:15:53.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-27T22:15:53.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-27T22:15:53.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-06-27T22:15:53.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-27T22:15:53.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-27T22:15:53.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-27T22:15:53.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:15:53.582 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:15:53.582 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:15:53.582 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:15:53.582 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:15:53.583 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:15:53.583 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:15:54.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-27T22:15:54.004 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:15:54.005 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:15:54.005 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:15:54.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-27T22:15:54.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-27T22:15:54.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-27T22:15:54.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-27T22:15:54.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-27T22:15:54.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-27T22:15:54.561 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-27T22:15:54.561 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-27T22:15:54.561 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-27T22:15:55.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-27T22:15:55.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-27T22:15:55.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-06-27T22:15:55.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-27T22:15:55.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-06-27T22:15:55.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-27T22:15:55.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:15:55.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:15:55.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:15:55.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:15:55.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:15:55.444 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:15:55.444 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:15:55.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-27T22:15:55.871 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:15:55.872 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:15:55.872 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:15:56.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-27T22:15:56.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-27T22:15:56.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-27T22:15:56.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-27T22:15:56.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-27T22:15:56.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-27T22:15:56.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-27T22:15:56.452 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-27T22:15:56.452 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-27T22:15:57.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-27T22:15:57.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-27T22:15:57.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-06-27T22:15:57.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-06-27T22:15:57.677 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T22:15:57.677 INFO:tasks.workunit.client.0.smithi042.stdout: "cluster_fingerprint": "f014cd92-2d47-4315-b7d5-6b705f789092", 2024-06-27T22:15:57.677 INFO:tasks.workunit.client.0.smithi042.stdout: "version": "19.0.0-4578-gf51acc18", 2024-06-27T22:15:57.677 INFO:tasks.workunit.client.0.smithi042.stdout: "commit": "f51acc18cc950586c82c58d666661d4391fc1122", 2024-06-27T22:15:57.677 INFO:tasks.workunit.client.0.smithi042.stdout: "timestamp": "2024-06-27T22:15:57.676906+0000", 2024-06-27T22:15:57.677 INFO:tasks.workunit.client.0.smithi042.stdout: "tag": "", 2024-06-27T22:15:57.677 INFO:tasks.workunit.client.0.smithi042.stdout: "health": { 2024-06-27T22:15:57.677 INFO:tasks.workunit.client.0.smithi042.stdout: "status": "HEALTH_WARN", 2024-06-27T22:15:57.677 INFO:tasks.workunit.client.0.smithi042.stdout: "checks": { 2024-06-27T22:15:57.677 INFO:tasks.workunit.client.0.smithi042.stdout: "PG_DEGRADED": { 2024-06-27T22:15:57.678 INFO:tasks.workunit.client.0.smithi042.stdout: "severity": "HEALTH_WARN", 2024-06-27T22:15:57.678 INFO:tasks.workunit.client.0.smithi042.stdout: "summary": { 2024-06-27T22:15:57.678 INFO:tasks.workunit.client.0.smithi042.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2024-06-27T22:15:57.678 INFO:tasks.workunit.client.0.smithi042.stdout: "count": 2 2024-06-27T22:15:57.678 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.678 INFO:tasks.workunit.client.0.smithi042.stdout: "detail": [ 2024-06-27T22:15:57.678 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.678 INFO:tasks.workunit.client.0.smithi042.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-06-27T22:15:57.678 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.678 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.678 INFO:tasks.workunit.client.0.smithi042.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: "muted": false 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: "TOO_FEW_OSDS": { 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: "severity": "HEALTH_WARN", 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: "summary": { 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: "count": 2 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: "detail": [], 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: "muted": false 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.679 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "mutes": [] 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "monmap_first_committed": 1, 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "monmap_last_committed": 1, 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "monmap": { 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "epoch": 1, 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "fsid": "0e761281-cb1c-448b-bd43-8b5efce19754", 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "modified": "2024-06-27T22:15:06.599312Z", 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "created": "2024-06-27T22:15:06.599312Z", 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "min_mon_release": 19, 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "min_mon_release_name": "squid", 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "election_strategy": 1, 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "disallowed_leaders: ": "", 2024-06-27T22:15:57.680 INFO:tasks.workunit.client.0.smithi042.stdout: "stretch_mode": false, 2024-06-27T22:15:57.681 INFO:tasks.workunit.client.0.smithi042.stdout: "tiebreaker_mon": "", 2024-06-27T22:15:57.681 INFO:tasks.workunit.client.0.smithi042.stdout: "removed_ranks: ": "", 2024-06-27T22:15:57.681 INFO:tasks.workunit.client.0.smithi042.stdout: "features": { 2024-06-27T22:15:57.681 INFO:tasks.workunit.client.0.smithi042.stdout: "persistent": [ 2024-06-27T22:15:57.681 INFO:tasks.workunit.client.0.smithi042.stdout: "kraken", 2024-06-27T22:15:57.681 INFO:tasks.workunit.client.0.smithi042.stdout: "luminous", 2024-06-27T22:15:57.681 INFO:tasks.workunit.client.0.smithi042.stdout: "mimic", 2024-06-27T22:15:57.681 INFO:tasks.workunit.client.0.smithi042.stdout: "osdmap-prune", 2024-06-27T22:15:57.681 INFO:tasks.workunit.client.0.smithi042.stdout: "nautilus", 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: "octopus", 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: "pacific", 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: "elector-pinging", 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: "quincy", 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: "reef", 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: "squid" 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: "optional": [] 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: "mons": [ 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.682 INFO:tasks.workunit.client.0.smithi042.stdout: "rank": 0, 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "a", 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: "public_addrs": { 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: "addrvec": [ 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v2", 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:7109", 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 0 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:7109/0", 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: "public_addr": "127.0.0.1:7109/0", 2024-06-27T22:15:57.683 INFO:tasks.workunit.client.0.smithi042.stdout: "priority": 0, 2024-06-27T22:15:57.684 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 0, 2024-06-27T22:15:57.684 INFO:tasks.workunit.client.0.smithi042.stdout: "crush_location": "{}" 2024-06-27T22:15:57.684 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.684 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.684 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.684 INFO:tasks.workunit.client.0.smithi042.stdout: "quorum": [ 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: 0 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: "osdmap": { 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: "epoch": 12, 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: "fsid": "0e761281-cb1c-448b-bd43-8b5efce19754", 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: "created": "2024-06-27T22:15:07.568030+0000", 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: "modified": "2024-06-27T22:15:44.688705+0000", 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: "last_up_change": "2024-06-27T22:15:40.968188+0000", 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: "last_in_change": "2024-06-27T22:15:08.344476+0000", 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: "flags_num": 5799936, 2024-06-27T22:15:57.685 INFO:tasks.workunit.client.0.smithi042.stdout: "flags_set": [ 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: "pglog_hardlimit", 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: "purged_snapdirs", 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: "recovery_deletes", 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: "sortbitwise" 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: "crush_version": 3, 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: "full_ratio": 0.99000000953674316, 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: "cluster_snapshot": "", 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: "pool_max": 1, 2024-06-27T22:15:57.686 INFO:tasks.workunit.client.0.smithi042.stdout: "max_osd": 1, 2024-06-27T22:15:57.687 INFO:tasks.workunit.client.0.smithi042.stdout: "require_min_compat_client": "luminous", 2024-06-27T22:15:57.687 INFO:tasks.workunit.client.0.smithi042.stdout: "min_compat_client": "jewel", 2024-06-27T22:15:57.687 INFO:tasks.workunit.client.0.smithi042.stdout: "require_osd_release": "squid", 2024-06-27T22:15:57.687 INFO:tasks.workunit.client.0.smithi042.stdout: "allow_crimson": false, 2024-06-27T22:15:57.687 INFO:tasks.workunit.client.0.smithi042.stdout: "pools": [ 2024-06-27T22:15:57.687 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.687 INFO:tasks.workunit.client.0.smithi042.stdout: "pool": 1, 2024-06-27T22:15:57.687 INFO:tasks.workunit.client.0.smithi042.stdout: "pool_name": "rbd", 2024-06-27T22:15:57.688 INFO:tasks.workunit.client.0.smithi042.stdout: "create_time": "2024-06-27T22:15:42.618054+0000", 2024-06-27T22:15:57.688 INFO:tasks.workunit.client.0.smithi042.stdout: "flags": 8193, 2024-06-27T22:15:57.688 INFO:tasks.workunit.client.0.smithi042.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-06-27T22:15:57.688 INFO:tasks.workunit.client.0.smithi042.stdout: "type": 1, 2024-06-27T22:15:57.688 INFO:tasks.workunit.client.0.smithi042.stdout: "size": 2, 2024-06-27T22:15:57.688 INFO:tasks.workunit.client.0.smithi042.stdout: "min_size": 1, 2024-06-27T22:15:57.689 INFO:tasks.workunit.client.0.smithi042.stdout: "crush_rule": 0, 2024-06-27T22:15:57.689 INFO:tasks.workunit.client.0.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-06-27T22:15:57.689 INFO:tasks.workunit.client.0.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-06-27T22:15:57.689 INFO:tasks.workunit.client.0.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T22:15:57.689 INFO:tasks.workunit.client.0.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T22:15:57.689 INFO:tasks.workunit.client.0.smithi042.stdout: "object_hash": 2, 2024-06-27T22:15:57.689 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_autoscale_mode": "off", 2024-06-27T22:15:57.689 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_num": 4, 2024-06-27T22:15:57.689 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_placement_num": 4, 2024-06-27T22:15:57.689 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_placement_num_target": 4, 2024-06-27T22:15:57.690 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_num_target": 4, 2024-06-27T22:15:57.690 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_num_pending": 4, 2024-06-27T22:15:57.690 INFO:tasks.workunit.client.0.smithi042.stdout: "last_pg_merge_meta": { 2024-06-27T22:15:57.690 INFO:tasks.workunit.client.0.smithi042.stdout: "source_pgid": "0.0", 2024-06-27T22:15:57.690 INFO:tasks.workunit.client.0.smithi042.stdout: "ready_epoch": 0, 2024-06-27T22:15:57.690 INFO:tasks.workunit.client.0.smithi042.stdout: "last_epoch_started": 0, 2024-06-27T22:15:57.690 INFO:tasks.workunit.client.0.smithi042.stdout: "last_epoch_clean": 0, 2024-06-27T22:15:57.690 INFO:tasks.workunit.client.0.smithi042.stdout: "source_version": "0'0", 2024-06-27T22:15:57.690 INFO:tasks.workunit.client.0.smithi042.stdout: "target_version": "0'0" 2024-06-27T22:15:57.691 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.691 INFO:tasks.workunit.client.0.smithi042.stdout: "last_change": "12", 2024-06-27T22:15:57.691 INFO:tasks.workunit.client.0.smithi042.stdout: "last_force_op_resend": "0", 2024-06-27T22:15:57.691 INFO:tasks.workunit.client.0.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T22:15:57.691 INFO:tasks.workunit.client.0.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-06-27T22:15:57.691 INFO:tasks.workunit.client.0.smithi042.stdout: "auid": 0, 2024-06-27T22:15:57.691 INFO:tasks.workunit.client.0.smithi042.stdout: "snap_mode": "selfmanaged", 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "snap_seq": 2, 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "snap_epoch": 12, 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "pool_snaps": [], 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "removed_snaps": "[]", 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "quota_max_bytes": 0, 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "quota_max_objects": 0, 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "tiers": [], 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "tier_of": -1, 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "read_tier": -1, 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "write_tier": -1, 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_mode": "none", 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "target_max_bytes": 0, 2024-06-27T22:15:57.692 INFO:tasks.workunit.client.0.smithi042.stdout: "target_max_objects": 0, 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_min_flush_age": 0, 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_min_evict_age": 0, 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "erasure_code_profile": "", 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "hit_set_params": { 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "none" 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "hit_set_period": 0, 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "hit_set_count": 0, 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "use_gmt_hitset": true, 2024-06-27T22:15:57.693 INFO:tasks.workunit.client.0.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "hit_set_search_last_n": 0, 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "grade_table": [], 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "stripe_width": 0, 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "expected_num_objects": 0, 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "fast_read": false, 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "options": {}, 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "application_metadata": { 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "rbd": {} 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "read_balance": { 2024-06-27T22:15:57.694 INFO:tasks.workunit.client.0.smithi042.stdout: "score_type": "Fair distribution", 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: "score_acting": 1, 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: "score_stable": 1, 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: "optimal_score": 1, 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: "raw_score_acting": 1, 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: "raw_score_stable": 1, 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: "primary_affinity_weighted": 1, 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: "average_primary_affinity": 1, 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: "osds": [ 2024-06-27T22:15:57.695 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "osd": 0, 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "uuid": "989af8a7-807b-4b46-a149-46eb5075cb94", 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "up": 1, 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "in": 1, 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 1, 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "primary_affinity": 1, 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "last_clean_begin": 0, 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "last_clean_end": 0, 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "up_from": 5, 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "up_thru": 7, 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "down_at": 0, 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "lost_at": 0, 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "public_addrs": { 2024-06-27T22:15:57.696 INFO:tasks.workunit.client.0.smithi042.stdout: "addrvec": [ 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v2", 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6800", 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 3805929884 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v1", 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6801", 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 3805929884 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: "cluster_addrs": { 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: "addrvec": [ 2024-06-27T22:15:57.697 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v2", 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6802", 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 3805929884 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v1", 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6803", 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 3805929884 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: "heartbeat_back_addrs": { 2024-06-27T22:15:57.698 INFO:tasks.workunit.client.0.smithi042.stdout: "addrvec": [ 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v2", 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6806", 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 3805929884 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v1", 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6807", 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 3805929884 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: "heartbeat_front_addrs": { 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: "addrvec": [ 2024-06-27T22:15:57.699 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.700 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v2", 2024-06-27T22:15:57.700 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6804", 2024-06-27T22:15:57.700 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 3805929884 2024-06-27T22:15:57.700 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.700 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.700 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v1", 2024-06-27T22:15:57.700 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6805", 2024-06-27T22:15:57.700 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 3805929884 2024-06-27T22:15:57.700 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.700 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: "public_addr": "127.0.0.1:6801/3805929884", 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: "cluster_addr": "127.0.0.1:6803/3805929884", 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: "heartbeat_back_addr": "127.0.0.1:6807/3805929884", 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: "heartbeat_front_addr": "127.0.0.1:6805/3805929884", 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: "state": [ 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: "exists", 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: "up" 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.701 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_xinfo": [ 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: "osd": 0, 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: "down_stamp": "0.000000", 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: "laggy_probability": 0, 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: "laggy_interval": 0, 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: "features": 4540701547738038271, 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: "old_weight": 0, 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: "last_purged_snaps_scrub": "2024-06-27T22:15:40.473379+0000", 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: "dead_epoch": 0 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_upmap": [], 2024-06-27T22:15:57.702 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_upmap_items": [], 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_upmap_primaries": [], 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_temp": [], 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "primary_temp": [], 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "blocklist": {}, 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "range_blocklist": {}, 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "erasure_code_profiles": { 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "default": { 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "k": "2", 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "m": "2", 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "plugin": "jerasure", 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "technique": "reed_sol_van" 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.703 INFO:tasks.workunit.client.0.smithi042.stdout: "removed_snaps_queue": [ 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: "pool": 1, 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: "snaps": [ 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: "begin": 2, 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: "length": 1 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: "new_removed_snaps": [ 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: "pool": 1, 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: "snaps": [ 2024-06-27T22:15:57.704 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: "begin": 2, 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: "length": 1 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: "new_purged_snaps": [], 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: "crush_node_flags": {}, 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: "device_class_flags": {}, 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: "stretch_mode": { 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: "stretch_mode_enabled": false, 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: "stretch_bucket_count": 0, 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: "degraded_stretch_mode": 0, 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: "recovering_stretch_mode": 0, 2024-06-27T22:15:57.705 INFO:tasks.workunit.client.0.smithi042.stdout: "stretch_mode_bucket": 0 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_metadata": [ 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: "id": 0, 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: "arch": "x86_64", 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: "back_addr": "[v2:127.0.0.1:6802/3805929884,v1:127.0.0.1:6803/3805929884]", 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: "back_iface": "", 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: "bluefs": "1", 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: "bluefs_dedicated_db": "0", 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: "bluefs_dedicated_wal": "0", 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: "bluefs_single_shared_device": "1", 2024-06-27T22:15:57.706 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_allocation_from_file": "0", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_access_mode": "file", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_block_size": "4096", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_devices": "sda", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_rotational": "1", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_size": "107374182400", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_support_discard": "0", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_type": "hdd", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_min_alloc_size": "4096", 2024-06-27T22:15:57.707 INFO:tasks.workunit.client.0.smithi042.stdout: "ceph_release": "squid", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "ceph_version": "ceph version 19.0.0-4578-gf51acc18 (f51acc18cc950586c82c58d666661d4391fc1122) squid (dev)", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "ceph_version_short": "19.0.0-4578-gf51acc18", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "ceph_version_when_created": "ceph version 19.0.0-4578-gf51acc18 (f51acc18cc950586c82c58d666661d4391fc1122) squid (dev)", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "created_at": "2024-06-27T22:15:14.045955Z", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "default_device_class": "hdd", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2MS", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "devices": "sda", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "distro": "ubuntu", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "distro_version": "22.04", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "front_addr": "[v2:127.0.0.1:6800/3805929884,v1:127.0.0.1:6801/3805929884]", 2024-06-27T22:15:57.708 INFO:tasks.workunit.client.0.smithi042.stdout: "front_iface": "", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/3805929884,v1:127.0.0.1:6807/3805929884]", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/3805929884,v1:127.0.0.1:6805/3805929884]", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "hostname": "smithi042", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "journal_rotational": "1", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "kernel_description": "#123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "kernel_version": "5.15.0-113-generic", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "mem_swap_kb": "2097148", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "mem_total_kb": "32753028", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "objectstore_numa_unknown_devices": "sda", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "os": "Linux", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_data": "td/ceph-helpers/0", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_objectstore": "bluestore", 2024-06-27T22:15:57.709 INFO:tasks.workunit.client.0.smithi042.stdout: "osdspec_affinity": "", 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: "rotational": "1" 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: "osdmap_clean_epochs": { 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: "min_last_epoch_clean": 0, 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: "last_epoch_clean": { 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: "per_pool": [] 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_epochs": [ 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: "id": 0, 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: "epoch": 5 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.710 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: "osdmap_first_committed": 1, 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: "osdmap_last_committed": 12, 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: "crushmap": { 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: "devices": [ 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: "id": 0, 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "osd.0", 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: "class": "hdd" 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: "types": [ 2024-06-27T22:15:57.711 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 0, 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "osd" 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 1, 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "host" 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 2, 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "chassis" 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 3, 2024-06-27T22:15:57.712 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "rack" 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 4, 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "row" 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 5, 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "pdu" 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 6, 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "pod" 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.713 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 7, 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "room" 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 8, 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "datacenter" 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 9, 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "zone" 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 10, 2024-06-27T22:15:57.714 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "region" 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 11, 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "root" 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: "buckets": [ 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -1, 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "default", 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 11, 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: "type_name": "root", 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: "alg": "straw2", 2024-06-27T22:15:57.715 INFO:tasks.workunit.client.0.smithi042.stdout: "hash": "rjenkins1", 2024-06-27T22:15:57.716 INFO:tasks.workunit.client.0.smithi042.stdout: "items": [ 2024-06-27T22:15:57.716 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.716 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -3, 2024-06-27T22:15:57.716 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:15:57.716 INFO:tasks.workunit.client.0.smithi042.stdout: "pos": 0 2024-06-27T22:15:57.716 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.716 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.716 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -2, 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "default~hdd", 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 11, 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: "type_name": "root", 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: "alg": "straw2", 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: "hash": "rjenkins1", 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: "items": [ 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -4, 2024-06-27T22:15:57.717 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: "pos": 0 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -3, 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "smithi042", 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 1, 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: "type_name": "host", 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: "alg": "straw2", 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: "hash": "rjenkins1", 2024-06-27T22:15:57.718 INFO:tasks.workunit.client.0.smithi042.stdout: "items": [ 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: "id": 0, 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: "pos": 0 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -4, 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "smithi042~hdd", 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 1, 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: "type_name": "host", 2024-06-27T22:15:57.719 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: "alg": "straw2", 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: "hash": "rjenkins1", 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: "items": [ 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: "id": 0, 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: "pos": 0 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: "rules": [ 2024-06-27T22:15:57.720 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: "rule_id": 0, 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: "rule_name": "replicated_rule", 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: "type": 1, 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: "steps": [ 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: "op": "take", 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: "item": -1, 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: "item_name": "default" 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: "op": "choose_firstn", 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: "num": 0, 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "osd" 2024-06-27T22:15:57.721 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: "op": "emit" 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: "tunables": { 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: "choose_local_tries": 0, 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: "choose_local_fallback_tries": 0, 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: "choose_total_tries": 50, 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: "chooseleaf_descend_once": 1, 2024-06-27T22:15:57.722 INFO:tasks.workunit.client.0.smithi042.stdout: "chooseleaf_vary_r": 1, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "chooseleaf_stable": 1, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "msr_descents": 100, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "msr_collision_tries": 100, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "straw_calc_version": 1, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "allowed_bucket_algs": 54, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "profile": "jewel", 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "optimal_tunables": 1, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "legacy_tunables": 0, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "minimum_required_version": "jewel", 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "require_feature_tunables": 1, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "require_feature_tunables2": 1, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "has_v2_rules": 0, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "require_feature_tunables3": 1, 2024-06-27T22:15:57.723 INFO:tasks.workunit.client.0.smithi042.stdout: "has_v3_rules": 0, 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "has_v4_buckets": 1, 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "require_feature_tunables5": 1, 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "has_v5_rules": 0, 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "has_msr_rules": 0 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "choose_args": {} 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "fsmap": { 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "epoch": 1, 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "btime": "2024-06-27T22:15:07:543494+0000", 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "default_fscid": -1, 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "compat": { 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "compat": {}, 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "ro_compat": {}, 2024-06-27T22:15:57.724 INFO:tasks.workunit.client.0.smithi042.stdout: "incompat": { 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_1": "base v0.20", 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_2": "client writeable ranges", 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_3": "default file layouts on dirs", 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_4": "dir inode in separate object", 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_5": "mds uses versioned encoding", 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_6": "dirfrag is stored in omap", 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_8": "no anchor table", 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_9": "file layout v2", 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_10": "snaprealm v2", 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_11": "minor log segments", 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_12": "quiesce subvolumes" 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_flags": { 2024-06-27T22:15:57.725 INFO:tasks.workunit.client.0.smithi042.stdout: "enable_multiple": true, 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: "ever_enabled_multiple": true 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: "standbys": [], 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: "filesystems": [] 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: "mdsmap_first_committed": 1, 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: "mdsmap_last_committed": 1, 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: "auth": { 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: "first_committed": 1, 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: "last_committed": 3, 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: "num_secrets": 1 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.726 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg": 4, 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg_active": 4, 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg_unknown": 0, 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "num_osd": 1, 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "pool_sum": { 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "stat_sum": { 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes": 19, 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects": 2, 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "num_object_clones": 0, 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "num_object_copies": 4, 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_missing_on_primary": 0, 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_missing": 0, 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_degraded": 2, 2024-06-27T22:15:57.727 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_misplaced": 0, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_unfound": 0, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_dirty": 2, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_whiteouts": 0, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_read": 0, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_read_kb": 0, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_write": 2, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_write_kb": 2, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_scrub_errors": 0, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_shallow_scrub_errors": 0, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_deep_scrub_errors": 0, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_recovered": 0, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes_recovered": 0, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_keys_recovered": 0, 2024-06-27T22:15:57.728 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_omap": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_hit_set_archive": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes_hit_set_archive": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_kb": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_kb": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_promote": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_mode_high": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_mode_low": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_mode_some": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_mode_full": 0, 2024-06-27T22:15:57.729 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_pinned": 0, 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "num_legacy_snapsets": 0, 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "num_large_omap_objects": 0, 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_manifest": 0, 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "num_omap_bytes": 0, 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "num_omap_keys": 0, 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_repaired": 0 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "store_stats": { 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "total": 0, 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "available": 0, 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "internally_reserved": 0, 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "allocated": 0, 2024-06-27T22:15:57.730 INFO:tasks.workunit.client.0.smithi042.stdout: "data_stored": 0, 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed": 0, 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_allocated": 0, 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_original": 0, 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "omap_allocated": 0, 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "internal_metadata": 0 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "log_size": 3, 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "ondisk_log_size": 3, 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "up": 4, 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "acting": 4, 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "num_store_stats": 0 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_sum": { 2024-06-27T22:15:57.731 INFO:tasks.workunit.client.0.smithi042.stdout: "up_from": 0, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "seq": 0, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pgs": 4, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "num_osds": 1, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "num_per_pool_osds": 1, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "num_per_pool_omap_osds": 1, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "kb": 104857600, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used": 26948, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_data": 108, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_omap": 1, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_meta": 26814, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_avail": 104830652, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "statfs": { 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "total": 107374182400, 2024-06-27T22:15:57.732 INFO:tasks.workunit.client.0.smithi042.stdout: "available": 107346587648, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "internally_reserved": 0, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "allocated": 110592, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "data_stored": 22938, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed": 0, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_allocated": 0, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_original": 0, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "omap_allocated": 1595, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "internal_metadata": 27457989 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "hb_peers": [], 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "snap_trim_queue_len": 0, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "num_snap_trimming": 0, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "num_shards_repaired": 0, 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "op_queue_age_hist": { 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "histogram": [], 2024-06-27T22:15:57.733 INFO:tasks.workunit.client.0.smithi042.stdout: "upper_bound": 1 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: "perf_stat": { 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: "commit_latency_ms": 0, 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: "apply_latency_ms": 0, 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: "commit_latency_ns": 0, 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: "apply_latency_ns": 0 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: "alerts": [], 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: "network_ping_times": [] 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_sum_by_class": { 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: "hdd": { 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: "up_from": 0, 2024-06-27T22:15:57.734 INFO:tasks.workunit.client.0.smithi042.stdout: "seq": 0, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pgs": 4, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "num_osds": 1, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "num_per_pool_osds": 1, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "num_per_pool_omap_osds": 1, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "kb": 104857600, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used": 26948, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_data": 108, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_omap": 1, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_meta": 26814, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_avail": 104830652, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "statfs": { 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "total": 107374182400, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "available": 107346587648, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "internally_reserved": 0, 2024-06-27T22:15:57.735 INFO:tasks.workunit.client.0.smithi042.stdout: "allocated": 110592, 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "data_stored": 22938, 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed": 0, 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_allocated": 0, 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_original": 0, 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "omap_allocated": 1595, 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "internal_metadata": 27457989 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "hb_peers": [], 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "snap_trim_queue_len": 0, 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "num_snap_trimming": 0, 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "num_shards_repaired": 0, 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "op_queue_age_hist": { 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "histogram": [], 2024-06-27T22:15:57.736 INFO:tasks.workunit.client.0.smithi042.stdout: "upper_bound": 1 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: "perf_stat": { 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: "commit_latency_ms": 0, 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: "apply_latency_ms": 0, 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: "commit_latency_ns": 0, 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: "apply_latency_ns": 0 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: "alerts": [], 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: "network_ping_times": [] 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: "pool_stats": [ 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: "poolid": 1, 2024-06-27T22:15:57.737 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg": 4, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "stat_sum": { 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes": 19, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects": 2, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_object_clones": 0, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_object_copies": 4, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_missing_on_primary": 0, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_missing": 0, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_degraded": 2, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_misplaced": 0, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_unfound": 0, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_dirty": 2, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_whiteouts": 0, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_read": 0, 2024-06-27T22:15:57.738 INFO:tasks.workunit.client.0.smithi042.stdout: "num_read_kb": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_write": 2, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_write_kb": 2, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_scrub_errors": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_shallow_scrub_errors": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_deep_scrub_errors": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_recovered": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes_recovered": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_keys_recovered": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_omap": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_hit_set_archive": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes_hit_set_archive": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_kb": 0, 2024-06-27T22:15:57.739 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_kb": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_promote": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_mode_high": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_mode_low": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_mode_some": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_mode_full": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_pinned": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_legacy_snapsets": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_large_omap_objects": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_manifest": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_omap_bytes": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_omap_keys": 0, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_repaired": 0 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.740 INFO:tasks.workunit.client.0.smithi042.stdout: "store_stats": { 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: "total": 0, 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: "available": 0, 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: "internally_reserved": 0, 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: "allocated": 4096, 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: "data_stored": 19, 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed": 0, 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_allocated": 0, 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_original": 0, 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: "omap_allocated": 0, 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: "internal_metadata": 0 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.741 INFO:tasks.workunit.client.0.smithi042.stdout: "log_size": 3, 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: "ondisk_log_size": 3, 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: "up": 4, 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: "acting": 4, 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: "num_store_stats": 1 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_stats": [ 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: "osd": 0, 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: "seq": 21474836482 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg_by_state": [ 2024-06-27T22:15:57.742 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: "state": "active+undersized+degraded", 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: "num": 2 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: "state": "active+undersized", 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: "num": 2 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg_by_osd": [ 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: "osd": 0, 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: "num_primary_pg": 4, 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: "num_acting_pg": 4, 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: "num_up_not_acting_pg": 0 2024-06-27T22:15:57.743 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: "purged_snaps": [ 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: "pool": 1, 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: "purged_snaps": {} 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: "servicemap": { 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: "epoch": 1, 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: "modified": "2024-06-27T22:15:07.568668+0000", 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: "services": {} 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:15:57.744 INFO:tasks.workunit.client.0.smithi042.stdout: "mgrstat_first_committed": 1, 2024-06-27T22:15:57.745 INFO:tasks.workunit.client.0.smithi042.stdout: "mgrstat_last_committed": 7, 2024-06-27T22:15:57.745 INFO:tasks.workunit.client.0.smithi042.stdout: "logm_first_committed": 1, 2024-06-27T22:15:57.745 INFO:tasks.workunit.client.0.smithi042.stdout: "logm_last_committed": 82, 2024-06-27T22:15:57.745 INFO:tasks.workunit.client.0.smithi042.stdout: "paxos": { 2024-06-27T22:15:57.745 INFO:tasks.workunit.client.0.smithi042.stdout: "first_committed": 1, 2024-06-27T22:15:57.745 INFO:tasks.workunit.client.0.smithi042.stdout: "last_committed": 108, 2024-06-27T22:15:57.745 INFO:tasks.workunit.client.0.smithi042.stdout: "last_pn": 0, 2024-06-27T22:15:57.745 INFO:tasks.workunit.client.0.smithi042.stdout: "accepted_pn": 0 2024-06-27T22:15:57.745 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:15:57.745 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T22:15:57.748 INFO:tasks.workunit.client.0.smithi042.stderr:report 1810613690 2024-06-27T22:15:57.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-06-27T22:15:57.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: test_wait_for_clean: run_osd td/ceph-helpers 1 2024-06-27T22:15:57.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:15:57.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:15:57.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:15:57.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:15:57.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-27T22:15:57.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:15:57.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:15:57.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:15:57.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9d1607f1-0fc6-4e59-9b92-ba38142282a5 2024-06-27T22:15:57.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9d1607f1-0fc6-4e59-9b92-ba38142282a5' 2024-06-27T22:15:57.751 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 9d1607f1-0fc6-4e59-9b92-ba38142282a5 2024-06-27T22:15:57.751 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:15:57.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCd5H1mGOUlKxAALYB276lPLJjKlNjg3gCC0Q== 2024-06-27T22:15:57.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCd5H1mGOUlKxAALYB276lPLJjKlNjg3gCC0Q=="}' 2024-06-27T22:15:57.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9d1607f1-0fc6-4e59-9b92-ba38142282a5 -i td/ceph-helpers/1/new.json 2024-06-27T22:15:58.214 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:15:58.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-27T22:15:58.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCd5H1mGOUlKxAALYB276lPLJjKlNjg3gCC0Q== --osd-uuid 9d1607f1-0fc6-4e59-9b92-ba38142282a5 2024-06-27T22:15:58.265 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:58.262+0000 7fd780edb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:15:58.275 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:58.274+0000 7fd780edb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:15:58.276 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:58.274+0000 7fd780edb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:15:58.276 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:58.274+0000 7fd780edb7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:15:58.277 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:58.274+0000 7fd780edb7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:15:58.277 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:58.274+0000 7fd780edb7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:15:58.306 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:15:58.306+0000 7fd780edb7c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-27T22:16:04.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-27T22:16:04.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:16:04.800 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:16:04.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:16:04.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:16:05.400 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:16:05.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:16:05.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:16:05.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:16:05.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:16:05.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:16:05.433 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:05.430+0000 7fa6e5bf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:16:05.436 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:05.434+0000 7fa6e5bf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:16:05.437 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:05.434+0000 7fa6e5bf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:16:05.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:16:05.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:16:05.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:16:05.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:16:05.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:16:05.866 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:16:05.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:05.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:16:05.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:05.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:05.992 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:05.990+0000 7fa6e5bf97c0 -1 Falling back to public interface 2024-06-27T22:16:06.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:07.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:07.324 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:07.324 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:16:07.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:16:07.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:07.325 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:07.773 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:08.666 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:08.666+0000 7fa6e5bf97c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:16:08.774 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:08.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:08.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:16:08.782 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:16:08.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:08.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:09.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:10.474 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:10.474 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:10.474 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:16:10.474 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:16:10.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:10.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:10.937 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:11.939 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:11.939 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:11.939 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:16:11.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:16:11.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:11.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:12.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:13.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:13.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:13.599 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:16:13.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:16:13.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:13.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:14.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:15.041 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:15.041 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:15.041 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:16:15.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:16:15.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:15.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:15.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:16.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:16.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:16.522 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:16:16.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:16:16.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:16.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:16.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:17.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:17.971 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:16:17.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:17.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:16:17.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:17.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:18.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:19.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:19.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:19.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:16:19.577 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:16:19.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:19.578 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:20.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:21.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:21.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:21.084 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:16:21.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:16:21.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:21.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:21.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:22.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:22.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:22.641 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:16:22.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:16:22.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:22.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:23.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:24.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:24.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:24.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:16:24.416 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:16:24.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:24.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:25.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:26.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:26.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:26.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:16:26.254 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:16:26.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:26.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:27.124 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:28.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:28.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:28.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:16:28.126 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:16:28.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:28.127 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:29.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:30.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:30.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:30.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:16:30.073 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:16:30.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:30.073 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:31.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:32.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:32.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:32.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:16:32.028 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:16:32.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:32.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:32.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:33.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:33.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:33.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:16:33.790 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:16:33.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:33.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:16:34.528 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1815645125,v1:127.0.0.1:6811/1815645125] [v2:127.0.0.1:6812/1815645125,v1:127.0.0.1:6813/1815645125] exists,up 9d1607f1-0fc6-4e59-9b92-ba38142282a5 2024-06-27T22:16:34.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:16:34.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:16:34.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:16:34.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1686: test_wait_for_clean: wait_for_clean 2024-06-27T22:16:34.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:16:34.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:16:34.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:16:34.530 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:16:34.530 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:16:34.530 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:16:34.530 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:16:34.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:16:34.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:16:34.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:16:34.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:16:34.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:16:34.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:16:34.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:16:34.639 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:16:35.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T22:16:35.142 INFO:tasks.workunit.client.0.smithi042.stderr:1' 2024-06-27T22:16:35.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:16:35.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:16:35.142 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:16:35.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836492 2024-06-27T22:16:35.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836492 2024-06-27T22:16:35.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492' 2024-06-27T22:16:35.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:16:35.371 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T22:16:35.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=68719476742 2024-06-27T22:16:35.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 68719476742 2024-06-27T22:16:35.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492 1-68719476742' 2024-06-27T22:16:35.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:16:35.555 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836492 2024-06-27T22:16:35.555 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:16:35.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:16:35.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836492 2024-06-27T22:16:35.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:16:35.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836492 2024-06-27T22:16:35.556 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836492 2024-06-27T22:16:35.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-06-27T22:16:35.557 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:16:36.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836490 -lt 21474836492 2024-06-27T22:16:36.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:16:37.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:16:37.022 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:16:37.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836492 2024-06-27T22:16:37.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:16:37.511 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-68719476742 2024-06-27T22:16:37.512 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:16:37.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T22:16:37.514 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-68719476742 2024-06-27T22:16:37.514 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:16:37.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=68719476742 2024-06-27T22:16:37.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 68719476742' 2024-06-27T22:16:37.516 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 68719476742 2024-06-27T22:16:37.516 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T22:16:37.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 68719476742 -lt 68719476742 2024-06-27T22:16:37.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:16:37.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:16:37.949 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:16:38.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:16:38.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:16:38.553 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:16:38.553 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:16:38.553 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:16:38.553 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:16:38.553 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:16:38.553 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:16:39.009 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-27T22:16:39.010 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:16:39.010 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:16:39.010 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:16:39.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 4 2024-06-27T22:16:39.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-27T22:16:39.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-27T22:16:39.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-27T22:16:39.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-27T22:16:39.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-27T22:16:39.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-27T22:16:39.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:16:39.670 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:16:39.670 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:16:39.670 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:16:39.670 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:16:39.670 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:16:39.670 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:16:40.090 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-27T22:16:40.091 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:16:40.091 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:16:40.091 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:16:40.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 4 2024-06-27T22:16:40.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-27T22:16:40.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-27T22:16:40.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-27T22:16:40.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-27T22:16:40.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-27T22:16:40.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-27T22:16:40.645 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-27T22:16:40.645 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-27T22:16:41.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-06-27T22:16:41.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-06-27T22:16:41.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-27T22:16:41.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-27T22:16:41.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-27T22:16:41.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-27T22:16:41.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:16:41.374 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:16:41.374 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:16:41.374 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:16:41.374 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:16:41.375 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:16:41.375 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:16:41.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:16:41.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:16:41.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:16:41.811 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:16:42.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:16:42.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:16:42.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:16:42.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: test_wait_for_clean: teardown td/ceph-helpers 2024-06-27T22:16:42.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:16:42.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:16:42.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:16:42.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:16:42.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:16:42.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:16:42.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:16:42.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:16:42.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:16:42.509 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:16:42.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:16:42.511 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:16:42.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:16:42.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:16:42.512 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:16:42.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:16:42.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:16:42.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:16:42.514 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:16:42.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:16:42.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:16:42.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:16:42.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:16:42.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:16:42.525 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:16:42.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:16:42.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:16:42.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:16:42.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:16:42.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_for_health_ok td/ceph-helpers 2024-06-27T22:16:42.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1822: test_wait_for_health_ok: local dir=td/ceph-helpers 2024-06-27T22:16:42.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1824: test_wait_for_health_ok: setup td/ceph-helpers 2024-06-27T22:16:42.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:16:42.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:16:42.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:16:42.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:16:42.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:16:42.528 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:16:42.528 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:16:42.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:16:42.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:16:42.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:16:42.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:16:42.529 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:16:42.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:16:42.530 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:16:42.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:16:42.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:16:42.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:16:42.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:16:42.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:16:42.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:16:42.533 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:16:42.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:16:42.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:16:42.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:16:42.536 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:16:42.536 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:16:42.536 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:16:42.536 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:16:42.536 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:16:42.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:16:42.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:16:42.538 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:16:42.538 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:16:42.538 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:16:42.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:16:42.540 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:16:42.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:16:42.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:16:42.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:16:42.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1825: test_wait_for_health_ok: run_mon td/ceph-helpers a --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-06-27T22:16:42.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:16:42.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:16:42.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:16:42.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:16:42.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:16:42.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-06-27T22:16:42.998 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:16:42.999 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:16:42.999 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:16:42.999 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:16:42.999 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:16:42.999 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:16:43.000 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:16:43.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-06-27T22:16:43.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:16:43.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:16:43.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:16:43.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:16:43.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:16:43.221 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:16:43.221 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:16:43.221 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:16:43.221 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:16:43.222 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:16:43.222 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:16:43.222 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:16:43.222 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:16:43.222 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:16:43.223 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:16:43.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:16:43.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:16:43.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:16:43.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:16:43.339 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:16:43.339 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:16:43.339 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:16:43.340 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:16:43.340 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:16:43.340 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:16:43.340 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:16:43.340 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:16:43.341 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:16:43.341 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:16:43.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1826: test_wait_for_health_ok: run_mgr td/ceph-helpers x --mon_pg_warn_min_per_osd=0 2024-06-27T22:16:43.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:16:43.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:16:43.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:16:43.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:16:43.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:16:43.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:16:44.378 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:16:44.379 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:16:44.379 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:16:44.379 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:16:44.379 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:16:44.379 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:16:44.379 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:16:44.379 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:16:44.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_pg_warn_min_per_osd=0 2024-06-27T22:16:44.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1828: test_wait_for_health_ok: run_osd td/ceph-helpers 0 2024-06-27T22:16:44.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:16:44.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:16:44.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:16:44.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:16:44.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:16:44.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:16:44.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:16:44.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:16:44.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:16:44.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:16:44.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:16:44.420 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:16:44.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=008f0394-eec7-40ae-a25f-226ed8df667f 2024-06-27T22:16:44.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 008f0394-eec7-40ae-a25f-226ed8df667f' 2024-06-27T22:16:44.421 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 008f0394-eec7-40ae-a25f-226ed8df667f 2024-06-27T22:16:44.421 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:16:44.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDM5H1mOds3GhAAT+yAIWy76nwmGLE+JC9mNA== 2024-06-27T22:16:44.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDM5H1mOds3GhAAT+yAIWy76nwmGLE+JC9mNA=="}' 2024-06-27T22:16:44.439 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 008f0394-eec7-40ae-a25f-226ed8df667f -i td/ceph-helpers/0/new.json 2024-06-27T22:16:44.730 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:16:44.746 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:16:44.747 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDM5H1mOds3GhAAT+yAIWy76nwmGLE+JC9mNA== --osd-uuid 008f0394-eec7-40ae-a25f-226ed8df667f 2024-06-27T22:16:44.782 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:44.781+0000 7fcd0a7d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:16:44.799 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:44.797+0000 7fcd0a7d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:16:44.800 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:44.797+0000 7fcd0a7d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:16:44.801 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:44.797+0000 7fcd0a7d07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:16:44.801 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:44.797+0000 7fcd0a7d07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:16:44.801 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:44.797+0000 7fcd0a7d07c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:16:44.839 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:44.837+0000 7fcd0a7d07c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:16:51.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:16:51.351 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:16:51.352 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:16:51.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:16:51.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:16:51.946 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:16:51.947 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:16:51.947 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:16:51.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:16:51.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:16:51.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:16:51.973 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:51.969+0000 7f4992c067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:16:52.005 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:52.001+0000 7f4992c067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:16:52.006 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:52.005+0000 7f4992c067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:16:52.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:16:52.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:16:52.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:16:52.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:16:52.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:16:52.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:52.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:16:52.408 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:16:52.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:52.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:16:52.595 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:52.593+0000 7f4992c067c0 -1 Falling back to public interface 2024-06-27T22:16:52.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:53.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:53.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:53.858 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:16:53.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:16:53.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:53.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:16:54.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:55.226 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:16:55.225+0000 7f4992c067c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:16:55.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:55.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:55.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:16:55.339 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:16:55.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:55.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:16:55.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:56.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:56.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:56.988 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:16:56.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:16:56.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:56.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:16:57.479 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:58.480 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:58.480 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:58.481 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:16:58.481 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:16:58.481 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:58.482 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:16:58.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:16:59.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:16:59.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:16:59.920 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:16:59.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:16:59.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:16:59.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:00.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:01.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:01.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:01.465 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:17:01.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:17:01.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:01.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:01.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:02.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:02.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:02.921 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:17:02.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:17:02.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:02.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:03.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:04.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:04.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:04.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:17:04.422 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:17:04.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:04.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:04.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:05.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:05.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:05.905 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:17:05.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:17:05.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:05.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:06.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:07.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:07.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:07.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:17:07.510 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:17:07.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:07.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:08.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:09.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:09.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:09.070 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:17:09.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:17:09.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:09.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:10.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:11.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:11.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:11.173 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:17:11.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:17:11.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:11.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:11.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:12.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:12.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:12.624 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:17:12.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:17:12.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:12.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:13.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:14.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:14.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:14.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:17:14.444 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:17:14.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:14.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:15.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:16.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:16.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:16.180 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:17:16.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:17:16.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:16.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:16.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:17.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:17.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:17.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:17:17.675 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:17:17.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:17.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:18.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:19.384 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:19.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:19.385 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:17:19.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:17:19.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:19.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:17:19.848 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/730230005,v1:127.0.0.1:6803/730230005] [v2:127.0.0.1:6804/730230005,v1:127.0.0.1:6805/730230005] exists,up 008f0394-eec7-40ae-a25f-226ed8df667f 2024-06-27T22:17:19.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:17:19.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:17:19.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1829: test_wait_for_health_ok: run_osd td/ceph-helpers 1 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-27T22:17:19.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:17:19.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:17:19.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:17:19.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:17:19.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:17:19.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:17:19.852 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:17:19.852 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:17:19.852 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:17:19.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:17:19.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:17:19.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:17:19.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:17:19.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:17:19.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:17:19.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:17:19.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:17:19.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:17:19.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:17:19.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:17:19.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:17:19.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:17:19.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-27T22:17:19.855 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:17:19.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=664dcdea-b93b-4c98-952f-03063dcaf5e2 2024-06-27T22:17:19.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 664dcdea-b93b-4c98-952f-03063dcaf5e2' 2024-06-27T22:17:19.856 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 664dcdea-b93b-4c98-952f-03063dcaf5e2 2024-06-27T22:17:19.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:17:19.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDv5H1mTOVxNBAAU03qrI89Snathqy+l9Ra+A== 2024-06-27T22:17:19.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDv5H1mTOVxNBAAU03qrI89Snathqy+l9Ra+A=="}' 2024-06-27T22:17:19.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 664dcdea-b93b-4c98-952f-03063dcaf5e2 -i td/ceph-helpers/1/new.json 2024-06-27T22:17:20.555 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:17:20.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-27T22:17:20.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDv5H1mTOVxNBAAU03qrI89Snathqy+l9Ra+A== --osd-uuid 664dcdea-b93b-4c98-952f-03063dcaf5e2 2024-06-27T22:17:20.615 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:20.613+0000 7f87c67917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:17:20.620 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:20.617+0000 7f87c67917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:17:20.621 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:20.617+0000 7f87c67917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:17:20.622 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:20.621+0000 7f87c67917c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:17:20.622 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:20.621+0000 7f87c67917c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:17:20.622 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:20.621+0000 7f87c67917c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:17:20.694 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:20.693+0000 7f87c67917c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-27T22:17:27.571 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-27T22:17:27.571 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:17:27.573 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:17:27.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:17:27.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:17:28.165 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:17:28.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:17:28.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:17:28.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:17:28.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:17:28.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:17:28.198 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:28.197+0000 7f98e72587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:17:28.227 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:28.225+0000 7f98e72587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:17:28.229 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:28.225+0000 7f98e72587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:17:28.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:17:28.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:17:28.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:17:28.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:17:28.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:17:28.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:28.634 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:17:28.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:17:28.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:28.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:28.795 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:28.793+0000 7f98e72587c0 -1 Falling back to public interface 2024-06-27T22:17:29.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:30.124 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:30.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:30.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:17:30.125 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:17:30.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:30.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:30.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:31.352 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:31.349+0000 7f98e72587c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:17:31.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:31.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:31.601 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:17:31.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:17:31.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:31.601 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:32.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:33.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:33.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:33.172 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:17:33.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:17:33.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:33.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:33.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:34.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:34.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:34.684 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:17:34.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:17:34.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:34.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:35.266 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:36.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:36.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:36.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:17:36.269 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:17:36.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:36.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:36.995 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:37.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:37.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:37.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:17:37.997 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:17:37.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:37.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:38.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:39.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:39.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:39.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:17:39.467 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:17:39.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:39.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:39.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:40.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:41.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:41.046 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:17:41.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:17:41.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:41.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:41.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:42.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:42.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:42.420 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:17:42.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:17:42.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:42.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:42.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:43.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:43.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:43.886 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:17:43.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:17:43.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:43.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:44.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:45.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:45.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:45.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:17:45.450 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:17:45.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:45.451 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:45.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:46.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:46.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:46.962 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:17:46.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:17:46.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:46.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:47.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:48.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:48.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:48.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:17:48.510 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:17:48.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:48.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:49.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:50.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:50.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:50.252 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:17:50.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:17:50.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:50.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:51.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:52.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:52.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:52.364 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:17:52.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:17:52.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:52.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:53.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:54.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:54.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:54.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:17:54.187 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:17:54.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:54.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:54.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:55.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:55.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:55.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:17:55.852 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:17:55.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:55.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:56.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:17:57.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:17:57.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:17:57.964 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:17:57.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:17:57.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:17:57.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:17:58.411 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1929614339,v1:127.0.0.1:6811/1929614339] [v2:127.0.0.1:6812/1929614339,v1:127.0.0.1:6813/1929614339] exists,up 664dcdea-b93b-4c98-952f-03063dcaf5e2 2024-06-27T22:17:58.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:17:58.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: test_wait_for_health_ok: run_osd td/ceph-helpers 2 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:17:58.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-06-27T22:17:58.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-06-27T22:17:58.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:17:58.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:17:58.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:17:58.414 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:17:58.414 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:17:58.414 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:17:58.414 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:17:58.414 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:17:58.414 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:17:58.415 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:17:58.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:17:58.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:17:58.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:17:58.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:17:58.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:17:58.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:17:58.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:17:58.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:17:58.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:17:58.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:17:58.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:17:58.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:17:58.417 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-06-27T22:17:58.417 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:17:58.417 INFO:tasks.workunit.client.0.smithi042.stdout:add osd2 5b062cda-7bbc-49dd-93fc-563ce08ab7ec 2024-06-27T22:17:58.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5b062cda-7bbc-49dd-93fc-563ce08ab7ec 2024-06-27T22:17:58.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5b062cda-7bbc-49dd-93fc-563ce08ab7ec' 2024-06-27T22:17:58.418 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:17:58.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAW5X1mwoNkGhAA71zOO2yAq0LRfR5CRmMTaQ== 2024-06-27T22:17:58.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAW5X1mwoNkGhAA71zOO2yAq0LRfR5CRmMTaQ=="}' 2024-06-27T22:17:58.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5b062cda-7bbc-49dd-93fc-563ce08ab7ec -i td/ceph-helpers/2/new.json 2024-06-27T22:17:59.081 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:17:59.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-06-27T22:17:59.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAW5X1mwoNkGhAA71zOO2yAq0LRfR5CRmMTaQ== --osd-uuid 5b062cda-7bbc-49dd-93fc-563ce08ab7ec 2024-06-27T22:17:59.136 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:59.133+0000 7fab93fec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:17:59.152 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:59.149+0000 7fab93fec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:17:59.153 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:59.149+0000 7fab93fec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:17:59.154 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:59.153+0000 7fab93fec7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-27T22:17:59.154 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:59.153+0000 7fab93fec7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-27T22:17:59.154 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:59.153+0000 7fab93fec7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-06-27T22:17:59.182 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:17:59.181+0000 7fab93fec7c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-06-27T22:18:05.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-06-27T22:18:05.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:18:05.844 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd2 key to auth repository 2024-06-27T22:18:05.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-27T22:18:05.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:18:06.441 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T22:18:06.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-27T22:18:06.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:18:06.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:18:06.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:18:06.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:18:06.473 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:06.473+0000 7feb82d8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:06.499 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:06.497+0000 7feb82d8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:06.501 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:06.501+0000 7feb82d8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:06.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-27T22:18:06.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:18:06.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T22:18:06.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:18:06.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:18:06.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:06.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:18:06.878 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:18:06.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:06.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:07.071 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:07.069+0000 7feb82d8e7c0 -1 Falling back to public interface 2024-06-27T22:18:07.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:08.356 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:08.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:08.357 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:18:08.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:18:08.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:08.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:08.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:09.793 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:09.793 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:09.793 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:18:09.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:18:09.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:09.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:09.827 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:09.825+0000 7feb82d8e7c0 -1 osd.2 0 log_to_monitors true 2024-06-27T22:18:10.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:11.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:11.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:11.432 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:18:11.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:18:11.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:11.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:11.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:12.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:12.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:12.914 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:18:12.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:18:12.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:12.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:13.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:14.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:14.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:14.382 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:18:14.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:18:14.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:14.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:14.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:15.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:15.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:15.985 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:18:15.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:18:15.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:15.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:16.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:17.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:17.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:17.424 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:18:17.425 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:18:17.425 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:17.426 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:18.023 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:19.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:19.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:19.025 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:18:19.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:18:19.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:19.025 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:19.497 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:20.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:20.499 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:20.499 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:18:20.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:18:20.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:20.500 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:21.055 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:22.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:22.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:22.057 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:18:22.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:18:22.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:22.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:22.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:23.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:23.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:23.529 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:18:23.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:18:23.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:23.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:24.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:25.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:25.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:25.149 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:18:25.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:18:25.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:25.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:25.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:26.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:26.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:26.738 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:18:26.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:18:26.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:26.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:27.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:28.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:28.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:28.806 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:18:28.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:18:28.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:28.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:29.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:30.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:30.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:30.759 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:18:30.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:18:30.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:30.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:31.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:32.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:32.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:32.509 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:18:32.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:18:32.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:32.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:33.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:34.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:34.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:34.271 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:18:34.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:18:34.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:34.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:34.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:35.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:35.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:35.724 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:18:35.724 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:18:35.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:35.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:36.200 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3173902181,v1:127.0.0.1:6819/3173902181] [v2:127.0.0.1:6820/3173902181,v1:127.0.0.1:6821/3173902181] exists,up 5b062cda-7bbc-49dd-93fc-563ce08ab7ec 2024-06-27T22:18:36.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:18:36.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:18:36.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:18:36.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1831: test_wait_for_health_ok: kill_daemons td/ceph-helpers TERM osd 2024-06-27T22:18:36.202 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:18:36.202 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:18:36.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:18:36.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:18:36.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:18:37.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:18:37.513 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: test_wait_for_health_ok: ceph osd down 0 2024-06-27T22:18:37.952 INFO:tasks.workunit.client.0.smithi042.stderr:osd.0 is already down. 2024-06-27T22:18:37.968 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: test_wait_for_health_ok: TIMEOUT=1 2024-06-27T22:18:37.968 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: test_wait_for_health_ok: wait_for_health_ok 2024-06-27T22:18:37.968 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: wait_for_health HEALTH_OK 2024-06-27T22:18:37.968 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_health: local grepstr=HEALTH_OK 2024-06-27T22:18:37.969 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: get_timeout_delays 1 .1 2024-06-27T22:18:37.969 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:18:37.969 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:18:37.969 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:18:37.970 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:18:37.970 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:18:38.003 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: delays=('0.1' '0.2' '0.4' '0.3') 2024-06-27T22:18:38.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: local -a delays 2024-06-27T22:18:38.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: wait_for_health: local -i loop=0 2024-06-27T22:18:38.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-06-27T22:18:38.004 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-06-27T22:18:38.570 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 0 >= 4 )) 2024-06-27T22:18:38.571 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.1 2024-06-27T22:18:38.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-06-27T22:18:38.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-06-27T22:18:38.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-06-27T22:18:39.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 1 >= 4 )) 2024-06-27T22:18:39.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.2 2024-06-27T22:18:39.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-06-27T22:18:39.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-06-27T22:18:39.437 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-06-27T22:18:40.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 2 >= 4 )) 2024-06-27T22:18:40.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.4 2024-06-27T22:18:40.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-06-27T22:18:40.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-06-27T22:18:40.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-06-27T22:18:41.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 3 >= 4 )) 2024-06-27T22:18:41.052 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.3 2024-06-27T22:18:41.353 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-06-27T22:18:41.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-06-27T22:18:41.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-06-27T22:18:41.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 4 >= 4 )) 2024-06-27T22:18:41.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1803: wait_for_health: ceph health detail 2024-06-27T22:18:42.524 INFO:tasks.workunit.client.0.smithi042.stdout:HEALTH_WARN 3 osds down; 1 host (3 osds) down; 1 root (3 osds) down 2024-06-27T22:18:42.525 INFO:tasks.workunit.client.0.smithi042.stdout:[WRN] OSD_DOWN: 3 osds down 2024-06-27T22:18:42.525 INFO:tasks.workunit.client.0.smithi042.stdout: osd.0 (root=default,host=smithi042) is down 2024-06-27T22:18:42.525 INFO:tasks.workunit.client.0.smithi042.stdout: osd.1 (root=default,host=smithi042) is down 2024-06-27T22:18:42.525 INFO:tasks.workunit.client.0.smithi042.stdout: osd.2 (root=default,host=smithi042) is down 2024-06-27T22:18:42.525 INFO:tasks.workunit.client.0.smithi042.stdout:[WRN] OSD_HOST_DOWN: 1 host (3 osds) down 2024-06-27T22:18:42.525 INFO:tasks.workunit.client.0.smithi042.stdout: host smithi042 (root=default) (3 osds) is down 2024-06-27T22:18:42.525 INFO:tasks.workunit.client.0.smithi042.stdout:[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down 2024-06-27T22:18:42.525 INFO:tasks.workunit.client.0.smithi042.stdout: root default (3 osds) is down 2024-06-27T22:18:42.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1804: wait_for_health: return 1 2024-06-27T22:18:42.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: return 1 2024-06-27T22:18:42.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: test_wait_for_health_ok: activate_osd td/ceph-helpers 0 2024-06-27T22:18:42.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-27T22:18:42.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-27T22:18:42.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-27T22:18:42.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-27T22:18:42.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:18:42.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:18:42.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:18:42.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:18:42.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:18:42.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:18:42.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:18:42.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-27T22:18:42.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-27T22:18:42.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:18:42.544 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-27T22:18:42.544 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:18:42.545 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:18:42.546 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:18:42.546 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:18:42.546 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:18:42.546 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:18:42.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:18:42.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:18:42.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:18:42.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:18:42.546 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:18:42.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:18:42.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:18:42.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:18:42.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-27T22:18:42.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-27T22:18:42.547 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:18:42.547 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:18:42.548 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-27T22:18:42.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:18:42.549 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-06-27T22:18:42.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-27T22:18:42.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-27T22:18:42.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-27T22:18:42.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-27T22:18:42.578 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:42.577+0000 7efe337237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:42.595 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:42.593+0000 7efe337237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:42.596 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:42.593+0000 7efe337237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:42.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-27T22:18:42.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:18:42.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:18:42.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:18:42.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:18:42.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:42.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:18:42.982 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:18:42.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:42.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:18:43.163 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:43.161+0000 7efe337237c0 -1 Falling back to public interface 2024-06-27T22:18:43.440 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:44.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:44.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:44.442 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:18:44.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:18:44.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:44.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:18:44.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:45.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:45.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:45.879 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:18:45.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:18:45.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:45.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:18:46.303 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:46.301+0000 7efe337237c0 -1 osd.0 16 log_to_monitors true 2024-06-27T22:18:46.318 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:47.320 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:47.320 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:18:47.321 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:47.321 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:18:47.321 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:47.322 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:18:47.796 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6802/2687704768,v1:127.0.0.1:6803/2687704768] [v2:127.0.0.1:6804/2687704768,v1:127.0.0.1:6805/2687704768] exists,up 008f0394-eec7-40ae-a25f-226ed8df667f 2024-06-27T22:18:47.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:18:47.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:18:47.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:18:47.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: test_wait_for_health_ok: activate_osd td/ceph-helpers 1 2024-06-27T22:18:47.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-27T22:18:47.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-27T22:18:47.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-27T22:18:47.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-27T22:18:47.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/1 2024-06-27T22:18:47.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:18:47.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:18:47.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:18:47.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:18:47.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-27T22:18:47.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-27T22:18:47.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-27T22:18:47.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-27T22:18:47.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:18:47.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-27T22:18:47.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:18:47.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:18:47.804 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:18:47.804 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:18:47.804 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:18:47.804 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:18:47.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:18:47.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:18:47.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:18:47.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:18:47.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:18:47.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:18:47.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:18:47.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:18:47.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-27T22:18:47.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-27T22:18:47.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/1 2024-06-27T22:18:47.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-27T22:18:47.805 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:18:47.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:18:47.806 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/1/whoami 2024-06-27T22:18:47.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-27T22:18:47.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-27T22:18:47.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-27T22:18:47.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-27T22:18:47.833 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:47.829+0000 7f40c7e047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:47.867 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:47.865+0000 7f40c7e047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:47.869 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:47.865+0000 7f40c7e047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:48.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-27T22:18:48.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:18:48.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:18:48.261 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:18:48.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:18:48.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:18:48.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:48.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:18:48.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:48.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:18:48.419 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:48.417+0000 7f40c7e047c0 -1 Falling back to public interface 2024-06-27T22:18:48.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:49.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:49.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:49.690 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:18:49.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:18:49.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:49.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:18:50.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:51.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:51.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:51.175 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:18:51.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:18:51.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:51.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:18:51.409 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:51.405+0000 7f40c7e047c0 -1 osd.1 16 log_to_monitors true 2024-06-27T22:18:51.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:52.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:52.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:52.703 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:18:52.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:18:52.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:52.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:18:53.202 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 21 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6810/3116568735,v1:127.0.0.1:6811/3116568735] [v2:127.0.0.1:6812/3116568735,v1:127.0.0.1:6813/3116568735] exists,up 664dcdea-b93b-4c98-952f-03063dcaf5e2 2024-06-27T22:18:53.204 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:18:53.204 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: test_wait_for_health_ok: activate_osd td/ceph-helpers 2 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/2 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-06-27T22:18:53.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-27T22:18:53.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-27T22:18:53.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:18:53.206 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-27T22:18:53.206 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:18:53.206 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:18:53.206 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:18:53.206 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:18:53.206 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:18:53.206 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:18:53.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:18:53.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:18:53.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:18:53.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:18:53.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:18:53.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:18:53.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:18:53.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:18:53.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-27T22:18:53.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-27T22:18:53.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/2 2024-06-27T22:18:53.208 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T22:18:53.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-27T22:18:53.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:18:53.209 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/2/whoami 2024-06-27T22:18:53.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-27T22:18:53.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-27T22:18:53.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-27T22:18:53.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-27T22:18:53.240 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:53.237+0000 7fb98b7b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:53.247 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:53.245+0000 7fb98b7b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:53.248 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:53.245+0000 7fb98b7b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:18:53.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-27T22:18:53.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:18:53.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T22:18:53.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:18:53.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:18:53.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:53.669 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:18:53.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:18:53.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:53.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:53.803 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:53.801+0000 7fb98b7b17c0 -1 Falling back to public interface 2024-06-27T22:18:54.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:55.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:55.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:55.113 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:18:55.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:18:55.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:55.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:55.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:56.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:56.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:56.588 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:18:56.589 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:18:56.589 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:56.589 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:56.814 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:18:56.813+0000 7fb98b7b17c0 -1 osd.2 16 log_to_monitors true 2024-06-27T22:18:57.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:18:58.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:18:58.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:18:58.044 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:18:58.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:18:58.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:18:58.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:18:58.493 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [15,16) [v2:127.0.0.1:6818/1674888973,v1:127.0.0.1:6819/1674888973] [v2:127.0.0.1:6820/1674888973,v1:127.0.0.1:6821/1674888973] exists,up 5b062cda-7bbc-49dd-93fc-563ce08ab7ec 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: test_wait_for_health_ok: wait_for_health_ok 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: wait_for_health HEALTH_OK 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_health: local grepstr=HEALTH_OK 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: get_timeout_delays 300 .1 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:18:58.495 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:18:58.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2024-06-27T22:18:58.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: local -a delays 2024-06-27T22:18:58.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: wait_for_health: local -i loop=0 2024-06-27T22:18:58.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-06-27T22:18:58.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-06-27T22:18:59.271 INFO:tasks.workunit.client.0.smithi042.stdout:HEALTH_OK 2024-06-27T22:18:59.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: test_wait_for_health_ok: teardown td/ceph-helpers 2024-06-27T22:18:59.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:18:59.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:18:59.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:18:59.273 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:18:59.273 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:18:59.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:18:59.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:18:59.274 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:18:59.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:18:59.586 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:18:59.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:18:59.587 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:18:59.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:18:59.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:18:59.589 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:18:59.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:18:59.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:18:59.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:18:59.591 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:18:59.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:18:59.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:18:59.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:18:59.613 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:18:59.613 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:18:59.613 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:18:59.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:18:59.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:18:59.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:18:59.615 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_for_osd td/ceph-helpers 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:995: test_wait_for_osd: local dir=td/ceph-helpers 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:996: test_wait_for_osd: setup td/ceph-helpers 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:18:59.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:18:59.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:18:59.618 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:18:59.619 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:18:59.619 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:18:59.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:18:59.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:18:59.621 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:18:59.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:18:59.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:18:59.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:18:59.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:18:59.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:18:59.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:18:59.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:18:59.626 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:18:59.626 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:18:59.627 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:18:59.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:18:59.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:18:59.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:18:59.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:18:59.629 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:18:59.629 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:18:59.629 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:18:59.629 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:18:59.632 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:18:59.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:18:59.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:18:59.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:18:59.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:997: test_wait_for_osd: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:18:59.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:18:59.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:18:59.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:18:59.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:18:59.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:18:59.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:19:00.096 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:19:00.096 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:19:00.096 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:19:00.097 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:19:00.097 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:19:00.097 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:19:00.097 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:19:00.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:19:00.315 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:19:00.316 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:19:00.316 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:19:00.316 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:19:00.316 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:19:00.317 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:19:00.317 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:19:00.317 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:19:00.317 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:19:00.317 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:19:00.317 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:19:00.318 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:19:00.318 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:19:00.318 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:19:00.318 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:19:00.436 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:19:00.437 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:19:00.437 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:19:00.437 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:19:00.438 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:19:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:19:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:19:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:19:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:19:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:19:00.438 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:19:00.439 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:19:00.439 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:19:00.439 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:19:00.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:998: test_wait_for_osd: run_mgr td/ceph-helpers x 2024-06-27T22:19:00.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:19:00.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:19:00.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:19:00.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:19:00.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:19:00.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:19:01.474 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:19:01.475 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:19:01.475 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:19:01.475 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:19:01.475 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:19:01.475 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:19:01.475 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:19:01.475 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:19:01.476 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:19:01.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:999: test_wait_for_osd: run_osd td/ceph-helpers 0 2024-06-27T22:19:01.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:19:01.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:19:01.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:19:01.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:19:01.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:19:01.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:19:01.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:19:01.560 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:19:01.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:19:01.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:19:01.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:19:01.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:19:01.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:19:01.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:19:01.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:19:01.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:19:01.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:19:01.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:19:01.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:19:01.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:19:01.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:19:01.563 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:19:01.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cda699ea-7b63-406d-9ce0-ff67d3d1342f 2024-06-27T22:19:01.563 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 cda699ea-7b63-406d-9ce0-ff67d3d1342f 2024-06-27T22:19:01.563 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cda699ea-7b63-406d-9ce0-ff67d3d1342f' 2024-06-27T22:19:01.563 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:19:01.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBV5X1mHbXhIhAAwoyC5tdCoieAtAdF5FVxAw== 2024-06-27T22:19:01.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBV5X1mHbXhIhAAwoyC5tdCoieAtAdF5FVxAw=="}' 2024-06-27T22:19:01.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cda699ea-7b63-406d-9ce0-ff67d3d1342f -i td/ceph-helpers/0/new.json 2024-06-27T22:19:01.920 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:19:01.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:19:01.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBV5X1mHbXhIhAAwoyC5tdCoieAtAdF5FVxAw== --osd-uuid cda699ea-7b63-406d-9ce0-ff67d3d1342f 2024-06-27T22:19:01.974 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:01.973+0000 7fbe597f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:01.997 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:01.997+0000 7fbe597f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:01.999 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:01.997+0000 7fbe597f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:01.999 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:01.997+0000 7fbe597f27c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:19:02.000 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:01.997+0000 7fbe597f27c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:19:02.000 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:01.997+0000 7fbe597f27c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:19:02.037 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:02.033+0000 7fbe597f27c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:19:08.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:19:08.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:19:08.646 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:19:08.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:19:08.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:19:09.247 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:19:09.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:19:09.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:19:09.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:19:09.248 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:19:09.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:19:09.281 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:09.277+0000 7f2d4271f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:09.305 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:09.305+0000 7f2d4271f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:09.307 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:09.305+0000 7f2d4271f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:09.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:19:09.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:19:09.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:19:09.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:19:09.704 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:19:09.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:19:09.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:09.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:19:09.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:09.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:09.909 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:09.905+0000 7f2d4271f7c0 -1 Falling back to public interface 2024-06-27T22:19:10.156 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:11.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:11.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:11.158 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:19:11.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:19:11.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:11.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:11.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:12.547 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:12.545+0000 7f2d4271f7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:19:12.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:12.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:12.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:19:12.639 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:19:12.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:12.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:13.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:14.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:14.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:14.283 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:19:14.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:19:14.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:14.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:14.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:15.773 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:15.773 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:15.773 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:19:15.774 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:19:15.774 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:15.774 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:16.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:17.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:17.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:17.219 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:19:17.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:19:17.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:17.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:17.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:18.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:18.696 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:18.696 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:19:18.696 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:19:18.696 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:18.696 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:19.155 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:20.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:20.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:20.157 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:19:20.158 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:19:20.158 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:20.158 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:20.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:21.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:21.705 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:21.706 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:19:21.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:19:21.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:21.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:22.156 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:23.157 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:23.158 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:23.158 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:19:23.158 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:19:23.158 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:23.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:23.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:24.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:24.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:24.757 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:19:24.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:19:24.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:24.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:25.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:26.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:26.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:26.214 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:19:26.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:19:26.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:26.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:27.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:28.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:28.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:28.038 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:19:28.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:19:28.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:28.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:29.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:30.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:30.230 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:30.231 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:19:30.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:19:30.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:30.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:31.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:32.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:32.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:32.056 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:19:32.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:19:32.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:32.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:33.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:34.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:34.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:34.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:19:34.079 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:19:34.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:34.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:35.101 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:36.103 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:36.103 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:36.103 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:19:36.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:19:36.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:36.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:36.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:37.747 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:37.747 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:37.748 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:19:37.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:19:37.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:37.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:19:38.197 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1541417085,v1:127.0.0.1:6803/1541417085] [v2:127.0.0.1:6804/1541417085,v1:127.0.0.1:6805/1541417085] exists,up cda699ea-7b63-406d-9ce0-ff67d3d1342f 2024-06-27T22:19:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:19:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:19:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:19:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1000: test_wait_for_osd: run_osd td/ceph-helpers 1 2024-06-27T22:19:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:19:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:19:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:19:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:19:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-27T22:19:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:19:38.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:19:38.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:19:38.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:19:38.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-27T22:19:38.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-27T22:19:38.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:19:38.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:19:38.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:19:38.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:19:38.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:19:38.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:19:38.201 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:19:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:19:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:19:38.202 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:19:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:19:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:19:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:19:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:19:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:19:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:19:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:19:38.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:19:38.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:19:38.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:19:38.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:19:38.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:19:38.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-27T22:19:38.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:19:38.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d94428aa-5627-4a46-84db-290f0478a59e 2024-06-27T22:19:38.204 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 d94428aa-5627-4a46-84db-290f0478a59e 2024-06-27T22:19:38.204 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d94428aa-5627-4a46-84db-290f0478a59e' 2024-06-27T22:19:38.204 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:19:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB65X1mctVJDRAAXzHynVfxeVD+zIKReAJhEQ== 2024-06-27T22:19:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB65X1mctVJDRAAXzHynVfxeVD+zIKReAJhEQ=="}' 2024-06-27T22:19:38.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d94428aa-5627-4a46-84db-290f0478a59e -i td/ceph-helpers/1/new.json 2024-06-27T22:19:38.732 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:19:38.749 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-27T22:19:38.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB65X1mctVJDRAAXzHynVfxeVD+zIKReAJhEQ== --osd-uuid d94428aa-5627-4a46-84db-290f0478a59e 2024-06-27T22:19:38.777 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:38.777+0000 7fd1c9eb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:38.835 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:38.833+0000 7fd1c9eb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:38.837 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:38.833+0000 7fd1c9eb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:38.838 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:38.837+0000 7fd1c9eb87c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:19:38.838 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:38.837+0000 7fd1c9eb87c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:19:38.838 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:38.837+0000 7fd1c9eb87c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:19:38.960 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:38.957+0000 7fd1c9eb87c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-27T22:19:45.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-27T22:19:45.520 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:19:45.521 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:19:45.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:19:45.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:19:46.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:19:46.115 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:19:46.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:19:46.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:19:46.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:19:46.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:19:46.148 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:46.145+0000 7f9bffe577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:46.166 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:46.165+0000 7f9bffe577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:46.167 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:46.165+0000 7f9bffe577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:19:46.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:19:46.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:19:46.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:19:46.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:19:46.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:19:46.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:46.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:19:46.558 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:19:46.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:46.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:19:46.728 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:46.725+0000 7f9bffe577c0 -1 Falling back to public interface 2024-06-27T22:19:46.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:47.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:47.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:47.972 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:19:47.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:19:47.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:47.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:19:48.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:49.325 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:19:49.321+0000 7f9bffe577c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:19:49.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:49.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:49.406 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:19:49.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:19:49.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:49.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:19:50.082 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:51.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:51.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:51.084 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:19:51.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:19:51.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:51.085 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:19:51.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:52.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:52.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:52.580 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:19:52.580 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:19:52.580 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:52.580 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:19:53.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:54.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:54.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:54.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:19:54.061 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:19:54.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:54.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:19:54.579 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:55.580 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:55.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:55.581 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:19:55.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:19:55.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:55.582 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:19:56.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:57.018 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:57.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:57.019 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:19:57.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:19:57.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:57.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:19:57.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:19:58.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:19:58.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:19:58.505 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:19:58.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:19:58.505 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:19:58.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:19:59.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:00.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:00.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:00.200 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:20:00.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:20:00.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:00.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:20:00.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:01.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:01.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:01.641 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:20:01.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:20:01.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:01.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:20:02.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:03.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:03.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:03.181 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:20:03.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:20:03.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:03.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:20:03.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:04.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:04.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:04.934 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:20:04.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:20:04.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:04.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:20:05.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:06.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:06.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:06.935 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:20:06.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:20:06.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:06.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:20:07.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:08.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:08.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:08.852 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:20:08.852 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:20:08.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:08.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:20:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:11.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:11.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:11.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:20:11.081 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:20:11.082 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:11.082 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:20:11.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:12.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:12.521 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:12.522 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:20:12.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:20:12.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:12.522 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:20:13.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:14.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:14.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:14.648 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:20:14.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:20:14.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:14.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:20:15.127 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3175217393,v1:127.0.0.1:6811/3175217393] [v2:127.0.0.1:6812/3175217393,v1:127.0.0.1:6813/3175217393] exists,up d94428aa-5627-4a46-84db-290f0478a59e 2024-06-27T22:20:15.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:20:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:20:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:20:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1001: test_wait_for_osd: wait_for_osd up 0 2024-06-27T22:20:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:20:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:20:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:20:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:20:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:15.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:20:15.130 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:20:15.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:15.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:15.669 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1541417085,v1:127.0.0.1:6803/1541417085] [v2:127.0.0.1:6804/1541417085,v1:127.0.0.1:6805/1541417085] exists,up cda699ea-7b63-406d-9ce0-ff67d3d1342f 2024-06-27T22:20:15.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:20:15.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:20:15.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:20:15.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1002: test_wait_for_osd: wait_for_osd up 1 2024-06-27T22:20:15.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:20:15.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:20:15.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:20:15.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:20:15.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:15.671 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:20:15.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:20:15.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:15.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:20:16.111 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3175217393,v1:127.0.0.1:6811/3175217393] [v2:127.0.0.1:6812/3175217393,v1:127.0.0.1:6813/3175217393] exists,up d94428aa-5627-4a46-84db-290f0478a59e 2024-06-27T22:20:16.112 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:20:16.112 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:20:16.112 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:20:16.112 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1003: test_wait_for_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-06-27T22:20:16.112 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:20:16.112 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:20:16.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:20:16.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:20:16.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:20:16.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:20:16.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1004: test_wait_for_osd: wait_for_osd down 0 2024-06-27T22:20:16.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-06-27T22:20:16.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:20:16.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:20:16.422 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:20:16.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:20:16.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:16.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:20:16.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:16.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-06-27T22:20:17.098 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 12 last_clean_interval [0,0) [v2:127.0.0.1:6802/1541417085,v1:127.0.0.1:6803/1541417085] [v2:127.0.0.1:6804/1541417085,v1:127.0.0.1:6805/1541417085] exists cda699ea-7b63-406d-9ce0-ff67d3d1342f 2024-06-27T22:20:17.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:20:17.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:20:17.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:20:17.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: TIMEOUT=1 2024-06-27T22:20:17.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: wait_for_osd up 0 2024-06-27T22:20:17.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:20:17.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:20:17.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:20:17.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:20:17.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-06-27T22:20:17.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:20:17.100 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:20:17.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:17.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:17.536 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:18.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:18.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-06-27T22:20:18.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 1 2024-06-27T22:20:18.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1006: test_wait_for_osd: teardown td/ceph-helpers 2024-06-27T22:20:18.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:20:18.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:20:18.539 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:20:18.539 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:20:18.539 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:20:18.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:20:18.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:20:18.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:20:18.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:20:18.650 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:20:18.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:20:18.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:20:18.652 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:20:18.652 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:20:18.652 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:20:18.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:20:18.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:20:18.653 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:20:18.653 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:20:18.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:20:18.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:20:18.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:20:18.665 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:20:18.666 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:20:18.666 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:20:18.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:20:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:20:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:20:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:20:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_for_peered td/ceph-helpers 2024-06-27T22:20:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: test_wait_for_peered: local dir=td/ceph-helpers 2024-06-27T22:20:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1736: test_wait_for_peered: setup td/ceph-helpers 2024-06-27T22:20:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:20:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:20:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:20:18.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:20:18.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:20:18.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:20:18.669 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:20:18.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:20:18.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:20:18.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:20:18.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:20:18.670 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:20:18.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:20:18.671 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:20:18.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:20:18.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:20:18.672 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:20:18.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:20:18.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:20:18.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:20:18.673 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:20:18.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:20:18.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:20:18.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:20:18.676 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:20:18.676 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:20:18.676 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:20:18.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:20:18.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:20:18.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:20:18.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:20:18.678 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:20:18.678 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:20:18.679 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:20:18.679 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:20:18.680 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:20:18.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:20:18.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:20:18.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:20:18.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1737: test_wait_for_peered: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-06-27T22:20:18.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:20:18.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:20:18.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:20:18.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:20:18.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:20:18.681 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-06-27T22:20:19.174 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:20:19.175 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:20:19.175 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:20:19.175 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:20:19.175 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:20:19.175 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:20:19.176 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:20:19.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-27T22:20:19.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:20:19.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:20:19.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:20:19.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:20:19.380 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:20:19.382 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:20:19.382 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:20:19.382 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:20:19.382 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:20:19.383 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:20:19.383 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:20:19.383 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:20:19.383 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:20:19.383 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:20:19.383 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:20:19.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:20:19.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:20:19.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:20:19.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:20:19.502 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:20:19.503 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:20:19.503 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:20:19.503 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:20:19.503 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:20:19.503 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:20:19.503 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:20:19.504 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:20:19.504 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:20:19.504 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:20:19.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: test_wait_for_peered: run_osd td/ceph-helpers 0 2024-06-27T22:20:19.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:20:19.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:20:19.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:20:19.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:20:19.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:20:19.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:20:19.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:20:19.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:20:19.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:20:19.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:20:19.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:20:19.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:20:19.619 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:20:19.619 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:20:19.619 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:20:19.619 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:20:19.619 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:20:19.620 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:20:19.620 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:20:19.620 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:20:19.620 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:20:19.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:20:19.620 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:20:19.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:20:19.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:20:19.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:20:19.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:20:19.621 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:20:19.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:20:19.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:20:19.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:20:19.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:20:19.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:20:19.622 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:20:19.623 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:20:19.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a6313cae-1e84-490d-9aa0-c83d4b04b291 2024-06-27T22:20:19.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a6313cae-1e84-490d-9aa0-c83d4b04b291' 2024-06-27T22:20:19.623 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 a6313cae-1e84-490d-9aa0-c83d4b04b291 2024-06-27T22:20:19.624 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:20:19.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCj5X1mKVX8JRAAGRj9H8pJiJJOT9dmMu6Kww== 2024-06-27T22:20:19.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCj5X1mKVX8JRAAGRj9H8pJiJJOT9dmMu6Kww=="}' 2024-06-27T22:20:19.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a6313cae-1e84-490d-9aa0-c83d4b04b291 -i td/ceph-helpers/0/new.json 2024-06-27T22:20:20.550 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:20:20.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:20:20.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCj5X1mKVX8JRAAGRj9H8pJiJJOT9dmMu6Kww== --osd-uuid a6313cae-1e84-490d-9aa0-c83d4b04b291 2024-06-27T22:20:20.606 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:20.604+0000 7f9f05a617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:20:20.618 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:20.616+0000 7f9f05a617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:20:20.620 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:20.616+0000 7f9f05a617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:20:20.620 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:20.616+0000 7f9f05a617c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:20:20.621 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:20.616+0000 7f9f05a617c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:20:20.621 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:20.616+0000 7f9f05a617c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:20:20.659 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:20.656+0000 7f9f05a617c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:20:27.011 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:20:27.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:20:27.013 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:20:27.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:20:27.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:20:27.266 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:20:27.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:20:27.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:20:27.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:20:27.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:20:27.267 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:20:27.296 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:27.292+0000 7f3fbcf487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:20:27.298 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:27.296+0000 7f3fbcf487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:20:27.299 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:27.296+0000 7f3fbcf487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:20:27.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:20:27.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:20:27.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:20:27.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:20:27.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:20:27.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:27.509 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:20:27.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:20:27.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:27.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:27.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:27.856 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:27.852+0000 7f3fbcf487c0 -1 Falling back to public interface 2024-06-27T22:20:28.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:28.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:28.736 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:20:28.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:20:28.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:28.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:28.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:29.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:29.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:29.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:20:29.981 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:20:29.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:29.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:30.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:30.569 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:20:30.564+0000 7f3fbcf487c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:20:31.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:31.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:31.224 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:20:31.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:20:31.225 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:31.225 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:31.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:32.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:32.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:32.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:20:32.672 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:20:32.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:32.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:32.917 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:33.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:33.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:33.919 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:20:33.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:20:33.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:33.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:34.165 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:35.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:35.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:35.167 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:20:35.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:20:35.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:35.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:35.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:36.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:36.414 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:36.414 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:20:36.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:20:36.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:36.415 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:36.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:37.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:37.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:37.659 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:20:37.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:20:37.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:37.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:37.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:38.902 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:38.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:38.903 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:20:38.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:20:38.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:38.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:39.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:40.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:40.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:40.151 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:20:40.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:20:40.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:40.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:40.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:41.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:41.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:41.397 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:20:41.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:20:41.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:41.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:41.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:42.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:42.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:42.643 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:20:42.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:20:42.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:42.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:42.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:43.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:43.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:43.889 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:20:43.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:20:43.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:43.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:44.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:45.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:45.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:45.137 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:20:45.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:20:45.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:45.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:45.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:46.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:46.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:46.382 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:20:46.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:20:46.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:46.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:46.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:47.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:47.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:47.630 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:20:47.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:20:47.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:47.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:47.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:48.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:48.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:48.877 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:20:48.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:20:48.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:48.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:49.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:50.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:50.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:50.121 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:20:50.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:20:50.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:50.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:50.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:51.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:51.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:51.362 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T22:20:51.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T22:20:51.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:51.363 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:51.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:52.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:52.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:52.606 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T22:20:52.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T22:20:52.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:52.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:52.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:20:53.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:20:53.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:20:53.855 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T22:20:53.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T22:20:53.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:20:53.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:20:54.206 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/874602760,v1:127.0.0.1:6801/874602760] [v2:127.0.0.1:6802/874602760,v1:127.0.0.1:6803/874602760] exists,up a6313cae-1e84-490d-9aa0-c83d4b04b291 2024-06-27T22:20:54.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:20:54.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:20:54.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:20:54.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1739: test_wait_for_peered: run_mgr td/ceph-helpers x 2024-06-27T22:20:54.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:20:54.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:20:54.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:20:54.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:20:54.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:20:54.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:20:54.554 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:20:54.555 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:20:54.555 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:20:54.555 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:20:54.555 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:20:54.555 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:20:54.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:20:54.556 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:20:54.558 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:20:54.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1740: test_wait_for_peered: create_rbd_pool 2024-06-27T22:20:54.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:20:54.828 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:20:54.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:20:54.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:20:55.230 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:20:55.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:20:56.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:20:57.354 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: WAIT_FOR_CLEAN_TIMEOUT=1 2024-06-27T22:20:57.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: wait_for_clean 2024-06-27T22:20:57.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:20:57.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:20:57.355 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:20:57.355 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-06-27T22:20:57.356 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:20:57.356 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:20:57.356 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:20:57.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:20:57.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:20:57.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-06-27T22:20:57.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:20:57.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:20:57.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:20:57.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:20:57.390 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:20:57.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:20:57.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:20:57.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:20:57.665 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:20:57.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=0 2024-06-27T22:20:57.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 0 2024-06-27T22:20:57.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-0' 2024-06-27T22:20:57.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:20:57.815 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:20:57.815 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-0 2024-06-27T22:20:57.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:20:57.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-0 2024-06-27T22:20:57.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:20:57.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=0 2024-06-27T22:20:57.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-06-27T22:20:57.818 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 0 2024-06-27T22:20:57.818 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:20:58.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 0 -lt 0 2024-06-27T22:20:58.036 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:20:58.036 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:20:58.037 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:20:58.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-06-27T22:20:58.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-06-27T22:20:59.272 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:20:59.272 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:20:59.272 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:20:59.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-06-27T22:20:59.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-06-27T22:21:00.505 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:21:00.506 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:21:00.506 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:21:01.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:21:01.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:21:01.082 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:21:01.082 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:21:01.082 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:21:01.083 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:21:01.083 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:21:01.083 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:21:01.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-27T22:21:01.519 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:21:01.519 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:21:01.519 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:21:02.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-27T22:21:02.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-27T22:21:02.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-27T22:21:02.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-27T22:21:02.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-27T22:21:02.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-27T22:21:02.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-27T22:21:02.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:21:02.194 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:21:02.194 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:21:02.195 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:21:02.195 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:21:02.195 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:21:02.195 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:21:02.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-27T22:21:02.618 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:21:02.618 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:21:02.618 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:21:03.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-27T22:21:03.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-27T22:21:03.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-27T22:21:03.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-27T22:21:03.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-27T22:21:03.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-27T22:21:03.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-27T22:21:03.227 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-27T22:21:03.227 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-27T22:21:03.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-27T22:21:03.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-27T22:21:03.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-06-27T22:21:03.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-27T22:21:03.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-27T22:21:04.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-27T22:21:04.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:21:04.022 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:21:04.022 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:21:04.022 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:21:04.022 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:21:04.023 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:21:04.023 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:21:04.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-27T22:21:04.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:21:04.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:21:04.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:21:04.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-27T22:21:04.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-27T22:21:04.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-27T22:21:04.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-27T22:21:04.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-27T22:21:04.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-27T22:21:04.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-27T22:21:04.995 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-27T22:21:04.995 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-27T22:21:05.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-27T22:21:05.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-27T22:21:05.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-06-27T22:21:05.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-27T22:21:05.577 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-27T22:21:05.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-27T22:21:05.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:21:05.978 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:21:05.978 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:21:05.978 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:21:05.978 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:21:05.978 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:21:05.978 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:21:06.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-27T22:21:06.421 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:21:06.421 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:21:06.421 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:21:06.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-27T22:21:06.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-27T22:21:06.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-27T22:21:06.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-27T22:21:06.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-27T22:21:06.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-27T22:21:06.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-27T22:21:06.999 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-27T22:21:06.999 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-27T22:21:07.589 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-27T22:21:07.589 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-27T22:21:07.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-06-27T22:21:07.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-27T22:21:07.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-06-27T22:21:07.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-27T22:21:07.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:21:07.891 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:21:07.892 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:21:07.892 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:21:07.892 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:21:07.892 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:21:07.893 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:21:08.321 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-27T22:21:08.322 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:21:08.322 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:21:08.322 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:21:08.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-06-27T22:21:08.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-27T22:21:08.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-27T22:21:08.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-27T22:21:08.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-27T22:21:08.894 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-27T22:21:08.895 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-27T22:21:08.895 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-27T22:21:08.895 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-27T22:21:09.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-27T22:21:09.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-27T22:21:09.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-06-27T22:21:09.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-06-27T22:21:10.009 INFO:tasks.workunit.client.0.smithi042.stdout:{ 2024-06-27T22:21:10.009 INFO:tasks.workunit.client.0.smithi042.stdout: "cluster_fingerprint": "dfad3432-6d4c-4d56-9f36-0298c852b6bc", 2024-06-27T22:21:10.009 INFO:tasks.workunit.client.0.smithi042.stdout: "version": "19.0.0-4578-gf51acc18", 2024-06-27T22:21:10.009 INFO:tasks.workunit.client.0.smithi042.stdout: "commit": "f51acc18cc950586c82c58d666661d4391fc1122", 2024-06-27T22:21:10.009 INFO:tasks.workunit.client.0.smithi042.stdout: "timestamp": "2024-06-27T22:21:10.010060+0000", 2024-06-27T22:21:10.009 INFO:tasks.workunit.client.0.smithi042.stdout: "tag": "", 2024-06-27T22:21:10.009 INFO:tasks.workunit.client.0.smithi042.stdout: "health": { 2024-06-27T22:21:10.009 INFO:tasks.workunit.client.0.smithi042.stdout: "status": "HEALTH_WARN", 2024-06-27T22:21:10.009 INFO:tasks.workunit.client.0.smithi042.stdout: "checks": { 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: "PG_DEGRADED": { 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: "severity": "HEALTH_WARN", 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: "summary": { 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: "count": 2 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: "detail": [ 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2024-06-27T22:21:10.010 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: "muted": false 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: "TOO_FEW_OSDS": { 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: "severity": "HEALTH_WARN", 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: "summary": { 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: "count": 2 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: "detail": [], 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: "muted": false 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.011 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "mutes": [] 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "monmap_first_committed": 1, 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "monmap_last_committed": 1, 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "monmap": { 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "epoch": 1, 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "fsid": "0e761281-cb1c-448b-bd43-8b5efce19754", 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "modified": "2024-06-27T22:20:18.706082Z", 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "created": "2024-06-27T22:20:18.706082Z", 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "min_mon_release": 19, 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "min_mon_release_name": "squid", 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "election_strategy": 1, 2024-06-27T22:21:10.012 INFO:tasks.workunit.client.0.smithi042.stdout: "disallowed_leaders: ": "", 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "stretch_mode": false, 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "tiebreaker_mon": "", 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "removed_ranks: ": "", 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "features": { 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "persistent": [ 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "kraken", 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "luminous", 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "mimic", 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "osdmap-prune", 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "nautilus", 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "octopus", 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "pacific", 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "elector-pinging", 2024-06-27T22:21:10.013 INFO:tasks.workunit.client.0.smithi042.stdout: "quincy", 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: "reef", 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: "squid" 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: "optional": [] 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: "mons": [ 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: "rank": 0, 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "a", 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: "public_addrs": { 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: "addrvec": [ 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.014 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v2", 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:7109", 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 0 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:7109/0", 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: "public_addr": "127.0.0.1:7109/0", 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: "priority": 0, 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 0, 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: "crush_location": "{}" 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.015 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "quorum": [ 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: 0 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "osdmap": { 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "epoch": 12, 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "fsid": "0e761281-cb1c-448b-bd43-8b5efce19754", 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "created": "2024-06-27T22:20:19.733837+0000", 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "modified": "2024-06-27T22:20:57.134870+0000", 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "last_up_change": "2024-06-27T22:20:52.955675+0000", 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "last_in_change": "2024-06-27T22:20:20.446835+0000", 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "flags_num": 5799936, 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "flags_set": [ 2024-06-27T22:21:10.016 INFO:tasks.workunit.client.0.smithi042.stdout: "pglog_hardlimit", 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "purged_snapdirs", 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "recovery_deletes", 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "sortbitwise" 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "crush_version": 3, 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "full_ratio": 0.99000000953674316, 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "cluster_snapshot": "", 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "pool_max": 1, 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "max_osd": 1, 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "require_min_compat_client": "luminous", 2024-06-27T22:21:10.017 INFO:tasks.workunit.client.0.smithi042.stdout: "min_compat_client": "jewel", 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "require_osd_release": "squid", 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "allow_crimson": false, 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "pools": [ 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "pool": 1, 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "pool_name": "rbd", 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "create_time": "2024-06-27T22:20:55.033665+0000", 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "flags": 8193, 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "type": 1, 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "size": 2, 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "min_size": 1, 2024-06-27T22:21:10.018 INFO:tasks.workunit.client.0.smithi042.stdout: "crush_rule": 0, 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "peering_crush_bucket_count": 0, 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "peering_crush_bucket_target": 0, 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "object_hash": 2, 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_autoscale_mode": "off", 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_num": 4, 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_placement_num": 4, 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_placement_num_target": 4, 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_num_target": 4, 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_num_pending": 4, 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "last_pg_merge_meta": { 2024-06-27T22:21:10.019 INFO:tasks.workunit.client.0.smithi042.stdout: "source_pgid": "0.0", 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "ready_epoch": 0, 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "last_epoch_started": 0, 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "last_epoch_clean": 0, 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "source_version": "0'0", 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "target_version": "0'0" 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "last_change": "12", 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "last_force_op_resend": "0", 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "last_force_op_resend_preluminous": "0", 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "auid": 0, 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "snap_mode": "selfmanaged", 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "snap_seq": 2, 2024-06-27T22:21:10.020 INFO:tasks.workunit.client.0.smithi042.stdout: "snap_epoch": 12, 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "pool_snaps": [], 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "removed_snaps": "[]", 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "quota_max_bytes": 0, 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "quota_max_objects": 0, 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "tiers": [], 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "tier_of": -1, 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "read_tier": -1, 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "write_tier": -1, 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_mode": "none", 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "target_max_bytes": 0, 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "target_max_objects": 0, 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T22:21:10.021 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_min_flush_age": 0, 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "cache_min_evict_age": 0, 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "erasure_code_profile": "", 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "hit_set_params": { 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "none" 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "hit_set_period": 0, 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "hit_set_count": 0, 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "use_gmt_hitset": true, 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "min_read_recency_for_promote": 0, 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "min_write_recency_for_promote": 0, 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T22:21:10.022 INFO:tasks.workunit.client.0.smithi042.stdout: "hit_set_search_last_n": 0, 2024-06-27T22:21:10.023 INFO:tasks.workunit.client.0.smithi042.stdout: "grade_table": [], 2024-06-27T22:21:10.023 INFO:tasks.workunit.client.0.smithi042.stdout: "stripe_width": 0, 2024-06-27T22:21:10.023 INFO:tasks.workunit.client.0.smithi042.stdout: "expected_num_objects": 0, 2024-06-27T22:21:10.023 INFO:tasks.workunit.client.0.smithi042.stdout: "fast_read": false, 2024-06-27T22:21:10.023 INFO:tasks.workunit.client.0.smithi042.stdout: "options": {}, 2024-06-27T22:21:10.023 INFO:tasks.workunit.client.0.smithi042.stdout: "application_metadata": { 2024-06-27T22:21:10.023 INFO:tasks.workunit.client.0.smithi042.stdout: "rbd": {} 2024-06-27T22:21:10.023 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.023 INFO:tasks.workunit.client.0.smithi042.stdout: "read_balance": { 2024-06-27T22:21:10.023 INFO:tasks.workunit.client.0.smithi042.stdout: "score_type": "Fair distribution", 2024-06-27T22:21:10.024 INFO:tasks.workunit.client.0.smithi042.stdout: "score_acting": 1, 2024-06-27T22:21:10.024 INFO:tasks.workunit.client.0.smithi042.stdout: "score_stable": 1, 2024-06-27T22:21:10.024 INFO:tasks.workunit.client.0.smithi042.stdout: "optimal_score": 1, 2024-06-27T22:21:10.024 INFO:tasks.workunit.client.0.smithi042.stdout: "raw_score_acting": 1, 2024-06-27T22:21:10.024 INFO:tasks.workunit.client.0.smithi042.stdout: "raw_score_stable": 1, 2024-06-27T22:21:10.024 INFO:tasks.workunit.client.0.smithi042.stdout: "primary_affinity_weighted": 1, 2024-06-27T22:21:10.024 INFO:tasks.workunit.client.0.smithi042.stdout: "average_primary_affinity": 1, 2024-06-27T22:21:10.024 INFO:tasks.workunit.client.0.smithi042.stdout: "average_primary_affinity_weighted": 1 2024-06-27T22:21:10.024 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: "osds": [ 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: "osd": 0, 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: "uuid": "a6313cae-1e84-490d-9aa0-c83d4b04b291", 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: "up": 1, 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: "in": 1, 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 1, 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: "primary_affinity": 1, 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: "last_clean_begin": 0, 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: "last_clean_end": 0, 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: "up_from": 5, 2024-06-27T22:21:10.025 INFO:tasks.workunit.client.0.smithi042.stdout: "up_thru": 7, 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: "down_at": 0, 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: "lost_at": 0, 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: "public_addrs": { 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: "addrvec": [ 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v2", 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6800", 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 874602760 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v1", 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6801", 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 874602760 2024-06-27T22:21:10.026 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: "cluster_addrs": { 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: "addrvec": [ 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v2", 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6802", 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 874602760 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v1", 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6803", 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 874602760 2024-06-27T22:21:10.027 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: "heartbeat_back_addrs": { 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: "addrvec": [ 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v2", 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6806", 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 874602760 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v1", 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6807", 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 874602760 2024-06-27T22:21:10.028 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: "heartbeat_front_addrs": { 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: "addrvec": [ 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v2", 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6804", 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 874602760 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "v1", 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: "addr": "127.0.0.1:6805", 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: "nonce": 874602760 2024-06-27T22:21:10.029 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: "public_addr": "127.0.0.1:6801/874602760", 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: "cluster_addr": "127.0.0.1:6803/874602760", 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: "heartbeat_back_addr": "127.0.0.1:6807/874602760", 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: "heartbeat_front_addr": "127.0.0.1:6805/874602760", 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: "state": [ 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: "exists", 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: "up" 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_xinfo": [ 2024-06-27T22:21:10.030 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "osd": 0, 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "down_stamp": "0.000000", 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "laggy_probability": 0, 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "laggy_interval": 0, 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "features": 4540701547738038271, 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "old_weight": 0, 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "last_purged_snaps_scrub": "2024-06-27T22:20:52.625523+0000", 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "dead_epoch": 0 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_upmap": [], 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_upmap_items": [], 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_upmap_primaries": [], 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "pg_temp": [], 2024-06-27T22:21:10.031 INFO:tasks.workunit.client.0.smithi042.stdout: "primary_temp": [], 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: "blocklist": {}, 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: "range_blocklist": {}, 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: "erasure_code_profiles": { 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: "default": { 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: "k": "2", 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: "m": "2", 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: "plugin": "jerasure", 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: "technique": "reed_sol_van" 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: "removed_snaps_queue": [ 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.032 INFO:tasks.workunit.client.0.smithi042.stdout: "pool": 1, 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: "snaps": [ 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: "begin": 2, 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: "length": 1 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: "new_removed_snaps": [ 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: "pool": 1, 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: "snaps": [ 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.033 INFO:tasks.workunit.client.0.smithi042.stdout: "begin": 2, 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: "length": 1 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: "new_purged_snaps": [], 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: "crush_node_flags": {}, 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: "device_class_flags": {}, 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: "stretch_mode": { 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: "stretch_mode_enabled": false, 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: "stretch_bucket_count": 0, 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: "degraded_stretch_mode": 0, 2024-06-27T22:21:10.034 INFO:tasks.workunit.client.0.smithi042.stdout: "recovering_stretch_mode": 0, 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: "stretch_mode_bucket": 0 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_metadata": [ 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: "id": 0, 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: "arch": "x86_64", 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: "back_addr": "[v2:127.0.0.1:6802/874602760,v1:127.0.0.1:6803/874602760]", 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: "back_iface": "", 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: "bluefs": "1", 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: "bluefs_dedicated_db": "0", 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: "bluefs_dedicated_wal": "0", 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: "bluefs_single_shared_device": "1", 2024-06-27T22:21:10.035 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_allocation_from_file": "0", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_access_mode": "file", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_block_size": "4096", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_devices": "sda", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_rotational": "1", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_size": "107374182400", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_support_discard": "0", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_bdev_type": "hdd", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "bluestore_min_alloc_size": "4096", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "ceph_release": "squid", 2024-06-27T22:21:10.036 INFO:tasks.workunit.client.0.smithi042.stdout: "ceph_version": "ceph version 19.0.0-4578-gf51acc18 (f51acc18cc950586c82c58d666661d4391fc1122) squid (dev)", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "ceph_version_short": "19.0.0-4578-gf51acc18", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "ceph_version_when_created": "ceph version 19.0.0-4578-gf51acc18 (f51acc18cc950586c82c58d666661d4391fc1122) squid (dev)", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "created_at": "2024-06-27T22:20:26.321509Z", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "default_device_class": "hdd", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4J2MS", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "devices": "sda", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "distro": "ubuntu", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "distro_version": "22.04", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "front_addr": "[v2:127.0.0.1:6800/874602760,v1:127.0.0.1:6801/874602760]", 2024-06-27T22:21:10.037 INFO:tasks.workunit.client.0.smithi042.stdout: "front_iface": "", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/874602760,v1:127.0.0.1:6807/874602760]", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/874602760,v1:127.0.0.1:6805/874602760]", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "hostname": "smithi042", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "journal_rotational": "1", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "kernel_description": "#123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "kernel_version": "5.15.0-113-generic", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "mem_swap_kb": "2097148", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "mem_total_kb": "32753028", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "objectstore_numa_unknown_devices": "sda", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "os": "Linux", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_data": "td/ceph-helpers/0", 2024-06-27T22:21:10.038 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_objectstore": "bluestore", 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: "osdspec_affinity": "", 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: "rotational": "1" 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: "osdmap_clean_epochs": { 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: "min_last_epoch_clean": 0, 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: "last_epoch_clean": { 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: "per_pool": [] 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_epochs": [ 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: "id": 0, 2024-06-27T22:21:10.039 INFO:tasks.workunit.client.0.smithi042.stdout: "epoch": 5 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: "osdmap_first_committed": 1, 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: "osdmap_last_committed": 12, 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: "crushmap": { 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: "devices": [ 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: "id": 0, 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "osd.0", 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: "class": "hdd" 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.040 INFO:tasks.workunit.client.0.smithi042.stdout: "types": [ 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 0, 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "osd" 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 1, 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "host" 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 2, 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "chassis" 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.041 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 3, 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "rack" 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 4, 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "row" 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 5, 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "pdu" 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 6, 2024-06-27T22:21:10.042 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "pod" 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 7, 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "room" 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 8, 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "datacenter" 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 9, 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "zone" 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.043 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.044 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 10, 2024-06-27T22:21:10.044 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "region" 2024-06-27T22:21:10.044 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.044 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.044 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 11, 2024-06-27T22:21:10.044 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "root" 2024-06-27T22:21:10.044 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.044 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.044 INFO:tasks.workunit.client.0.smithi042.stdout: "buckets": [ 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -1, 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "default", 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 11, 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: "type_name": "root", 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: "alg": "straw2", 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: "hash": "rjenkins1", 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: "items": [ 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -3, 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: "pos": 0 2024-06-27T22:21:10.045 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -2, 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "default~hdd", 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 11, 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: "type_name": "root", 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: "alg": "straw2", 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: "hash": "rjenkins1", 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: "items": [ 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -4, 2024-06-27T22:21:10.046 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:21:10.047 INFO:tasks.workunit.client.0.smithi042.stdout: "pos": 0 2024-06-27T22:21:10.047 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.047 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.047 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.047 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.047 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -3, 2024-06-27T22:21:10.047 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "smithi042", 2024-06-27T22:21:10.047 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 1, 2024-06-27T22:21:10.047 INFO:tasks.workunit.client.0.smithi042.stdout: "type_name": "host", 2024-06-27T22:21:10.047 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: "alg": "straw2", 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: "hash": "rjenkins1", 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: "items": [ 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: "id": 0, 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: "pos": 0 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: "id": -4, 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: "name": "smithi042~hdd", 2024-06-27T22:21:10.048 INFO:tasks.workunit.client.0.smithi042.stdout: "type_id": 1, 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: "type_name": "host", 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: "alg": "straw2", 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: "hash": "rjenkins1", 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: "items": [ 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: "id": 0, 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: "weight": 6402, 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: "pos": 0 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.049 INFO:tasks.workunit.client.0.smithi042.stdout: "rules": [ 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: "rule_id": 0, 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: "rule_name": "replicated_rule", 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: "type": 1, 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: "steps": [ 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: "op": "take", 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: "item": -1, 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: "item_name": "default" 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: "op": "choose_firstn", 2024-06-27T22:21:10.050 INFO:tasks.workunit.client.0.smithi042.stdout: "num": 0, 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: "type": "osd" 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: "op": "emit" 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: ] 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: "tunables": { 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: "choose_local_tries": 0, 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: "choose_local_fallback_tries": 0, 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: "choose_total_tries": 50, 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: "chooseleaf_descend_once": 1, 2024-06-27T22:21:10.051 INFO:tasks.workunit.client.0.smithi042.stdout: "chooseleaf_vary_r": 1, 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "chooseleaf_stable": 1, 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "msr_descents": 100, 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "msr_collision_tries": 100, 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "straw_calc_version": 1, 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "allowed_bucket_algs": 54, 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "profile": "jewel", 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "optimal_tunables": 1, 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "legacy_tunables": 0, 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "minimum_required_version": "jewel", 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "require_feature_tunables": 1, 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "require_feature_tunables2": 1, 2024-06-27T22:21:10.052 INFO:tasks.workunit.client.0.smithi042.stdout: "has_v2_rules": 0, 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "require_feature_tunables3": 1, 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "has_v3_rules": 0, 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "has_v4_buckets": 1, 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "require_feature_tunables5": 1, 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "has_v5_rules": 0, 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "has_msr_rules": 0 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "choose_args": {} 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "fsmap": { 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "epoch": 1, 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "btime": "2024-06-27T22:20:19:701312+0000", 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "default_fscid": -1, 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "compat": { 2024-06-27T22:21:10.053 INFO:tasks.workunit.client.0.smithi042.stdout: "compat": {}, 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "ro_compat": {}, 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "incompat": { 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_1": "base v0.20", 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_2": "client writeable ranges", 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_3": "default file layouts on dirs", 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_4": "dir inode in separate object", 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_5": "mds uses versioned encoding", 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_6": "dirfrag is stored in omap", 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_8": "no anchor table", 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_9": "file layout v2", 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_10": "snaprealm v2", 2024-06-27T22:21:10.054 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_11": "minor log segments", 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_12": "quiesce subvolumes" 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: "feature_flags": { 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: "enable_multiple": true, 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: "ever_enabled_multiple": true 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: "standbys": [], 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: "filesystems": [] 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: "mdsmap_first_committed": 1, 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: "mdsmap_last_committed": 1, 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: "auth": { 2024-06-27T22:21:10.055 INFO:tasks.workunit.client.0.smithi042.stdout: "first_committed": 1, 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "last_committed": 3, 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "num_secrets": 1 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg": 4, 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg_active": 4, 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg_unknown": 0, 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "num_osd": 1, 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "pool_sum": { 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "stat_sum": { 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes": 19, 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects": 2, 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "num_object_clones": 0, 2024-06-27T22:21:10.056 INFO:tasks.workunit.client.0.smithi042.stdout: "num_object_copies": 4, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_missing_on_primary": 0, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_missing": 0, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_degraded": 2, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_misplaced": 0, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_unfound": 0, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_dirty": 2, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_whiteouts": 0, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_read": 0, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_read_kb": 0, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_write": 2, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_write_kb": 2, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_scrub_errors": 0, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_shallow_scrub_errors": 0, 2024-06-27T22:21:10.057 INFO:tasks.workunit.client.0.smithi042.stdout: "num_deep_scrub_errors": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_recovered": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes_recovered": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_keys_recovered": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_omap": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_hit_set_archive": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes_hit_set_archive": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_kb": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_kb": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_promote": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_mode_high": 0, 2024-06-27T22:21:10.058 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_mode_low": 0, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_mode_some": 0, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_mode_full": 0, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_pinned": 0, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "num_legacy_snapsets": 0, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "num_large_omap_objects": 0, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_manifest": 0, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "num_omap_bytes": 0, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "num_omap_keys": 0, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_repaired": 0 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "store_stats": { 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "total": 0, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "available": 0, 2024-06-27T22:21:10.059 INFO:tasks.workunit.client.0.smithi042.stdout: "internally_reserved": 0, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "allocated": 0, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "data_stored": 0, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed": 0, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_allocated": 0, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_original": 0, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "omap_allocated": 0, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "internal_metadata": 0 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "log_size": 3, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "ondisk_log_size": 3, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "up": 4, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "acting": 4, 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: "num_store_stats": 0 2024-06-27T22:21:10.060 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_sum": { 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "up_from": 0, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "seq": 0, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pgs": 4, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "num_osds": 1, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "num_per_pool_osds": 1, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "num_per_pool_omap_osds": 1, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "kb": 104857600, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used": 26948, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_data": 108, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_omap": 1, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_meta": 26814, 2024-06-27T22:21:10.061 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_avail": 104830652, 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "statfs": { 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "total": 107374182400, 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "available": 107346587648, 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "internally_reserved": 0, 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "allocated": 110592, 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "data_stored": 22938, 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed": 0, 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_allocated": 0, 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_original": 0, 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "omap_allocated": 1595, 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "internal_metadata": 27457989 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.062 INFO:tasks.workunit.client.0.smithi042.stdout: "hb_peers": [], 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: "snap_trim_queue_len": 0, 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: "num_snap_trimming": 0, 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: "num_shards_repaired": 0, 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: "op_queue_age_hist": { 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: "histogram": [], 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: "upper_bound": 1 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: "perf_stat": { 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: "commit_latency_ms": 0, 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: "apply_latency_ms": 0, 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: "commit_latency_ns": 0, 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: "apply_latency_ns": 0 2024-06-27T22:21:10.063 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "alerts": [], 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "network_ping_times": [] 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_sum_by_class": { 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "hdd": { 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "up_from": 0, 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "seq": 0, 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pgs": 4, 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "num_osds": 1, 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "num_per_pool_osds": 1, 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "num_per_pool_omap_osds": 1, 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "kb": 104857600, 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used": 26948, 2024-06-27T22:21:10.064 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_data": 108, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_omap": 1, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_used_meta": 26814, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "kb_avail": 104830652, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "statfs": { 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "total": 107374182400, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "available": 107346587648, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "internally_reserved": 0, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "allocated": 110592, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "data_stored": 22938, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed": 0, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_allocated": 0, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_original": 0, 2024-06-27T22:21:10.065 INFO:tasks.workunit.client.0.smithi042.stdout: "omap_allocated": 1595, 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "internal_metadata": 27457989 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "hb_peers": [], 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "snap_trim_queue_len": 0, 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "num_snap_trimming": 0, 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "num_shards_repaired": 0, 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "op_queue_age_hist": { 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "histogram": [], 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "upper_bound": 1 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "perf_stat": { 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "commit_latency_ms": 0, 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "apply_latency_ms": 0, 2024-06-27T22:21:10.066 INFO:tasks.workunit.client.0.smithi042.stdout: "commit_latency_ns": 0, 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: "apply_latency_ns": 0 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: "alerts": [], 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: "network_ping_times": [] 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: "pool_stats": [ 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: "poolid": 1, 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg": 4, 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: "stat_sum": { 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes": 19, 2024-06-27T22:21:10.067 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects": 2, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_object_clones": 0, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_object_copies": 4, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_missing_on_primary": 0, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_missing": 0, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_degraded": 2, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_misplaced": 0, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_unfound": 0, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_dirty": 2, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_whiteouts": 0, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_read": 0, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_read_kb": 0, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_write": 2, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_write_kb": 2, 2024-06-27T22:21:10.068 INFO:tasks.workunit.client.0.smithi042.stdout: "num_scrub_errors": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_shallow_scrub_errors": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_deep_scrub_errors": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_recovered": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes_recovered": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_keys_recovered": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_omap": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_hit_set_archive": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_bytes_hit_set_archive": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_kb": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_kb": 0, 2024-06-27T22:21:10.069 INFO:tasks.workunit.client.0.smithi042.stdout: "num_promote": 0, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_mode_high": 0, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "num_flush_mode_low": 0, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_mode_some": 0, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "num_evict_mode_full": 0, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_pinned": 0, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "num_legacy_snapsets": 0, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "num_large_omap_objects": 0, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_manifest": 0, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "num_omap_bytes": 0, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "num_omap_keys": 0, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "num_objects_repaired": 0 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.070 INFO:tasks.workunit.client.0.smithi042.stdout: "store_stats": { 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "total": 0, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "available": 0, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "internally_reserved": 0, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "allocated": 4096, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "data_stored": 19, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed": 0, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_allocated": 0, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "data_compressed_original": 0, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "omap_allocated": 0, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "internal_metadata": 0 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "log_size": 3, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "ondisk_log_size": 3, 2024-06-27T22:21:10.071 INFO:tasks.workunit.client.0.smithi042.stdout: "up": 4, 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: "acting": 4, 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: "num_store_stats": 1 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: "osd_stats": [ 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: "osd": 0, 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: "seq": 21474836482 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg_by_state": [ 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: "state": "active+undersized+degraded", 2024-06-27T22:21:10.072 INFO:tasks.workunit.client.0.smithi042.stdout: "num": 2 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: "state": "active+undersized", 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: "num": 2 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: "num_pg_by_osd": [ 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: "osd": 0, 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: "num_primary_pg": 4, 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: "num_acting_pg": 4, 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: "num_up_not_acting_pg": 0 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.073 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: "purged_snaps": [ 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: { 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: "pool": 1, 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: "purged_snaps": {} 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: ], 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: "servicemap": { 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: "epoch": 1, 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: "modified": "2024-06-27T22:20:19.734505+0000", 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: "services": {} 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: }, 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: "mgrstat_first_committed": 1, 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: "mgrstat_last_committed": 7, 2024-06-27T22:21:10.074 INFO:tasks.workunit.client.0.smithi042.stdout: "logm_first_committed": 1, 2024-06-27T22:21:10.075 INFO:tasks.workunit.client.0.smithi042.stdout: "logm_last_committed": 82, 2024-06-27T22:21:10.075 INFO:tasks.workunit.client.0.smithi042.stdout: "paxos": { 2024-06-27T22:21:10.075 INFO:tasks.workunit.client.0.smithi042.stdout: "first_committed": 1, 2024-06-27T22:21:10.075 INFO:tasks.workunit.client.0.smithi042.stdout: "last_committed": 108, 2024-06-27T22:21:10.075 INFO:tasks.workunit.client.0.smithi042.stdout: "last_pn": 0, 2024-06-27T22:21:10.075 INFO:tasks.workunit.client.0.smithi042.stdout: "accepted_pn": 0 2024-06-27T22:21:10.075 INFO:tasks.workunit.client.0.smithi042.stdout: } 2024-06-27T22:21:10.075 INFO:tasks.workunit.client.0.smithi042.stdout:} 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:report 1463257865 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: test_wait_for_peered: run_osd td/ceph-helpers 1 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:21:10.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:21:10.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-06-27T22:21:10.080 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:21:10.081 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 83725ca6-6ad3-4164-9d94-47b0e36f4df6 2024-06-27T22:21:10.082 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=83725ca6-6ad3-4164-9d94-47b0e36f4df6 2024-06-27T22:21:10.082 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 83725ca6-6ad3-4164-9d94-47b0e36f4df6' 2024-06-27T22:21:10.082 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:21:10.082 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDW5X1mIRjXAhAAyhltxuTsUBm3P87XrIPtxA== 2024-06-27T22:21:10.082 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDW5X1mIRjXAhAAyhltxuTsUBm3P87XrIPtxA=="}' 2024-06-27T22:21:10.082 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 83725ca6-6ad3-4164-9d94-47b0e36f4df6 -i td/ceph-helpers/1/new.json 2024-06-27T22:21:10.555 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:21:10.571 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-06-27T22:21:10.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDW5X1mIRjXAhAAyhltxuTsUBm3P87XrIPtxA== --osd-uuid 83725ca6-6ad3-4164-9d94-47b0e36f4df6 2024-06-27T22:21:10.607 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:10.604+0000 7f7ad014b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:21:10.611 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:10.608+0000 7f7ad014b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:21:10.612 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:10.608+0000 7f7ad014b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:21:10.613 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:10.608+0000 7f7ad014b7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:21:10.613 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:10.608+0000 7f7ad014b7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:21:10.613 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:10.608+0000 7f7ad014b7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-06-27T22:21:10.652 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:10.648+0000 7f7ad014b7c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-06-27T22:21:17.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-06-27T22:21:17.114 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:21:17.116 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:21:17.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:21:17.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:21:17.713 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:21:17.714 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:21:17.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:21:17.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:21:17.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:21:17.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:21:17.739 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:17.736+0000 7f53369447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:21:17.776 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:17.772+0000 7f53369447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:21:17.778 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:17.776+0000 7f53369447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:21:18.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:21:18.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:21:18.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:21:18.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:21:18.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:21:18.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:18.152 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:21:18.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:21:18.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:18.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:18.355 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:18.352+0000 7f53369447c0 -1 Falling back to public interface 2024-06-27T22:21:18.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:19.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:19.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:19.599 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:21:19.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:21:19.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:19.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:20.075 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:20.894 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:20.892+0000 7f53369447c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:21:21.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:21.077 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:21.077 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:21:21.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:21:21.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:21.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:21.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:22.689 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:22.689 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:22.690 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:21:22.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:21:22.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:22.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:23.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:24.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:24.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:24.306 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:21:24.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:21:24.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:24.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:24.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:25.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:25.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:25.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:21:25.771 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:21:25.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:25.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:26.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:27.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:27.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:27.207 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:21:27.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:21:27.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:27.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:27.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:28.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:28.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:28.809 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:21:28.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:21:28.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:28.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:29.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:30.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:30.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:30.271 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:21:30.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:21:30.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:30.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:30.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:31.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:31.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:31.790 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:21:31.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:21:31.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:31.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:32.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:33.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:33.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:33.236 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:21:33.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:21:33.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:33.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:33.773 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:34.775 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:34.775 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:34.775 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:21:34.775 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:21:34.776 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:34.776 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:35.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:36.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:36.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:36.435 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:21:36.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:21:36.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:36.436 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:37.438 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:38.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:38.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:38.443 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:21:38.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:21:38.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:38.443 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:39.013 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:40.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:40.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:40.015 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:21:40.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:21:40.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:40.016 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:40.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:41.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:41.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:41.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:21:41.851 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:21:41.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:41.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:43.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:44.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:44.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:44.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:21:44.037 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:21:44.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:44.038 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:45.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:21:46.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:21:46.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:21:46.210 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:21:46.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:21:46.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:21:46.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:21:46.869 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 16 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/991741900,v1:127.0.0.1:6811/991741900] [v2:127.0.0.1:6812/991741900,v1:127.0.0.1:6813/991741900] exists,up 83725ca6-6ad3-4164-9d94-47b0e36f4df6 2024-06-27T22:21:46.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:21:46.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:21:46.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:21:46.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: test_wait_for_peered: wait_for_peered 2024-06-27T22:21:46.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-06-27T22:21:46.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-06-27T22:21:46.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-06-27T22:21:46.871 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-06-27T22:21:46.872 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:21:46.872 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:21:46.872 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:21:46.872 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:21:46.872 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:21:46.988 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:21:46.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-06-27T22:21:46.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-06-27T22:21:46.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-06-27T22:21:46.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:21:46.989 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:21:47.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T22:21:47.518 INFO:tasks.workunit.client.0.smithi042.stderr:1' 2024-06-27T22:21:47.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:21:47.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:21:47.518 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:21:47.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836492 2024-06-27T22:21:47.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836492 2024-06-27T22:21:47.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492' 2024-06-27T22:21:47.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:21:47.777 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T22:21:47.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=68719476743 2024-06-27T22:21:47.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 68719476743 2024-06-27T22:21:47.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492 1-68719476743' 2024-06-27T22:21:47.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:21:47.935 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836492 2024-06-27T22:21:47.935 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:21:47.936 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:21:47.937 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836492 2024-06-27T22:21:47.937 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:21:47.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836492 2024-06-27T22:21:47.938 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-06-27T22:21:47.939 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836492 2024-06-27T22:21:47.939 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:21:48.476 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836490 -lt 21474836492 2024-06-27T22:21:48.476 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:21:49.478 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:21:49.478 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:21:49.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836492 2024-06-27T22:21:49.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:21:49.922 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-68719476743 2024-06-27T22:21:49.922 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:21:49.922 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T22:21:49.924 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:21:49.924 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-68719476743 2024-06-27T22:21:49.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=68719476743 2024-06-27T22:21:49.925 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 68719476743' 2024-06-27T22:21:49.925 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 68719476743 2024-06-27T22:21:49.926 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T22:21:50.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 68719476743 -lt 68719476743 2024-06-27T22:21:50.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-06-27T22:21:50.363 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:21:50.364 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:21:50.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 4 == 0 2024-06-27T22:21:50.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-06-27T22:21:50.941 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-06-27T22:21:50.941 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-06-27T22:21:50.941 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-06-27T22:21:50.941 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-06-27T22:21:50.941 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-06-27T22:21:50.941 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-06-27T22:21:51.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=4 2024-06-27T22:21:51.361 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-06-27T22:21:51.362 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:21:51.362 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:21:51.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 4 = 4 2024-06-27T22:21:51.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-06-27T22:21:51.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-06-27T22:21:51.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1744: test_wait_for_peered: teardown td/ceph-helpers 2024-06-27T22:21:51.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:21:51.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:21:51.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:21:51.933 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:21:51.933 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:21:51.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:21:51.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:21:51.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:21:52.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:21:52.041 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:21:52.041 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:21:52.041 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:21:52.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:21:52.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:21:52.042 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:21:52.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:21:52.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:21:52.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:21:52.044 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:21:52.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:21:52.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:21:52.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:21:52.055 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:21:52.055 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:21:52.056 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:21:52.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:21:52.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:21:52.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:21:52.057 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_for_scrub td/ceph-helpers 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: test_wait_for_scrub: local dir=td/ceph-helpers 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: test_wait_for_scrub: setup td/ceph-helpers 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:21:52.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:21:52.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:21:52.059 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:21:52.059 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:21:52.059 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:21:52.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:21:52.060 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:21:52.061 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:21:52.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:21:52.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:21:52.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:21:52.062 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:21:52.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:21:52.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:21:52.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:21:52.064 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:21:52.064 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:21:52.064 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:21:52.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:21:52.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:21:52.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:21:52.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:21:52.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:21:52.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:21:52.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:21:52.067 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:21:52.069 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:21:52.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:21:52.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:21:52.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:21:52.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2042: test_wait_for_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:21:52.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:21:52.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:21:52.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:21:52.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:21:52.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:21:52.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:21:52.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:21:52.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:21:52.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:21:52.567 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:21:52.567 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:21:52.567 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:21:52.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:21:52.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:21:52.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:21:52.787 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:21:52.787 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:21:52.787 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:21:52.787 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:21:52.788 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:21:52.788 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:21:52.788 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:21:52.788 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:21:52.788 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:21:52.789 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:21:52.789 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:21:52.789 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:21:52.789 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:21:52.789 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:21:52.904 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:21:52.904 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:21:52.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:21:52.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:21:52.906 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:21:52.906 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:21:52.906 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:21:52.906 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:21:52.906 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:21:52.906 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:21:52.906 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:21:52.906 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:21:52.907 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:21:52.907 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:21:53.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2043: test_wait_for_scrub: run_mgr td/ceph-helpers x 2024-06-27T22:21:53.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:21:53.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:21:53.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:21:53.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:21:53.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:21:53.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:21:53.917 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:21:53.918 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:21:53.918 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:21:53.918 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:21:53.918 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:21:53.918 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:21:53.918 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:21:53.918 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:21:53.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:21:53.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2044: test_wait_for_scrub: run_osd td/ceph-helpers 0 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:21:53.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:21:53.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:21:53.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:21:53.959 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:21:53.959 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:21:53.959 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:21:53.959 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:21:53.959 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:21:53.959 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:21:53.959 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:21:53.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:21:53.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:21:53.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:21:53.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:21:53.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:21:53.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:21:53.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:21:53.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:21:53.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:21:53.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:21:53.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:21:53.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:21:53.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:21:53.961 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:21:53.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd194e7f-6ae2-4ddb-af7c-066915035ad4 2024-06-27T22:21:53.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fd194e7f-6ae2-4ddb-af7c-066915035ad4' 2024-06-27T22:21:53.961 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 fd194e7f-6ae2-4ddb-af7c-066915035ad4 2024-06-27T22:21:53.961 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:21:53.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAB5n1m6ullOhAAMAbIwYD4J3uC+kDA8WnS9w== 2024-06-27T22:21:53.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAB5n1m6ullOhAAMAbIwYD4J3uC+kDA8WnS9w=="}' 2024-06-27T22:21:53.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd194e7f-6ae2-4ddb-af7c-066915035ad4 -i td/ceph-helpers/0/new.json 2024-06-27T22:21:54.298 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:21:54.315 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:21:54.316 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAB5n1m6ullOhAAMAbIwYD4J3uC+kDA8WnS9w== --osd-uuid fd194e7f-6ae2-4ddb-af7c-066915035ad4 2024-06-27T22:21:54.351 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:54.352+0000 7f2f80fbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:21:54.396 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:54.396+0000 7f2f80fbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:21:54.398 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:54.400+0000 7f2f80fbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:21:54.399 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:54.400+0000 7f2f80fbe7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:21:54.399 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:54.400+0000 7f2f80fbe7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:21:54.399 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:54.400+0000 7f2f80fbe7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:21:54.453 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:21:54.452+0000 7f2f80fbe7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:22:01.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:22:01.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:22:01.062 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:22:01.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:22:01.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:22:01.661 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:22:01.661 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:22:01.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:22:01.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:22:01.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:22:01.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:22:01.693 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:01.692+0000 7fa1759ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:22:01.698 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:01.700+0000 7fa1759ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:22:01.699 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:01.700+0000 7fa1759ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:22:02.091 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:22:02.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:22:02.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:22:02.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:22:02.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:22:02.092 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:02.093 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:22:02.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:22:02.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:02.093 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:02.293 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:02.292+0000 7fa1759ae7c0 -1 Falling back to public interface 2024-06-27T22:22:02.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:03.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:03.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:03.551 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:22:03.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:22:03.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:03.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:03.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:04.831 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:04.832+0000 7fa1759ae7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:22:04.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:04.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:04.999 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:22:04.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:22:04.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:04.999 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:05.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:06.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:06.613 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:22:06.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:06.613 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:22:06.614 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:06.614 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:07.068 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:08.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:08.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:08.070 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:22:08.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:22:08.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:08.071 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:08.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:09.663 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:09.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:09.664 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:22:09.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:22:09.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:09.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:10.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:11.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:11.129 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:11.129 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:22:11.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:22:11.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:11.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:11.662 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:12.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:12.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:12.664 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:22:12.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:22:12.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:12.665 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:13.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:14.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:14.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:14.130 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:22:14.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:22:14.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:14.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:14.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:15.721 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:15.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:15.722 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:22:15.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:22:15.722 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:15.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:16.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:17.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:17.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:17.190 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:22:17.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:22:17.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:17.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:17.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:18.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:18.644 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:18.644 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:22:18.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:22:18.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:18.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:19.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:20.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:20.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:20.236 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:22:20.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:22:20.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:20.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:21.026 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:22.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:22.028 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:22.028 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:22:22.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:22:22.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:22.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:22.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:23.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:23.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:23.858 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:22:23.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:22:23.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:23.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:24.570 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:25.571 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:25.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:25.572 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:22:25.572 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:22:25.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:25.573 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:26.753 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:27.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:27.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:27.755 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:22:27.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:22:27.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:27.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:28.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:29.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:29.730 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:29.731 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:22:29.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:22:29.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:29.731 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:30.178 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2865797258,v1:127.0.0.1:6803/2865797258] [v2:127.0.0.1:6804/2865797258,v1:127.0.0.1:6805/2865797258] exists,up fd194e7f-6ae2-4ddb-af7c-066915035ad4 2024-06-27T22:22:30.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:22:30.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:22:30.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:22:30.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: test_wait_for_scrub: create_rbd_pool 2024-06-27T22:22:30.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:22:30.609 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:22:30.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:22:30.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:22:31.157 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:22:31.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:22:32.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:22:33.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: test_wait_for_scrub: wait_for_clean 2024-06-27T22:22:33.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:22:33.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:22:33.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:22:33.391 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:22:33.391 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:22:33.391 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:22:33.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:22:33.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:22:33.392 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:22:33.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:22:33.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:22:33.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:22:33.523 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:22:33.524 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:22:33.524 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:22:33.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:22:33.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:22:33.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:22:33.960 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:22:34.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836486 2024-06-27T22:22:34.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836486 2024-06-27T22:22:34.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836486' 2024-06-27T22:22:34.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:22:34.120 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836486 2024-06-27T22:22:34.120 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:22:34.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:22:34.121 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836486 2024-06-27T22:22:34.122 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:22:34.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836486 2024-06-27T22:22:34.122 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-06-27T22:22:34.123 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836486 2024-06-27T22:22:34.123 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:22:34.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836484 -lt 21474836486 2024-06-27T22:22:34.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:22:35.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:22:35.551 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:22:36.002 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836486 -lt 21474836486 2024-06-27T22:22:36.003 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:22:36.003 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:22:36.003 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:22:36.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:22:36.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:22:36.586 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:22:36.587 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:22:36.587 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:22:36.587 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:22:36.587 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:22:36.587 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:22:37.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:22:37.052 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:22:37.052 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:22:37.052 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:22:37.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:22:37.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:22:37.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:22:37.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: test_wait_for_scrub: local pgid=1.0 2024-06-27T22:22:37.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: test_wait_for_scrub: ceph pg repair 1.0 2024-06-27T22:22:38.021 INFO:tasks.workunit.client.0.smithi042.stderr:instructing pg 1.0 on osd.0 to repair 2024-06-27T22:22:38.038 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-06-27T22:22:38.038 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:22:38.038 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:22:38.038 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:22:38.038 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:22:38.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: test_wait_for_scrub: local last_scrub=2024-06-27T22:22:31.006777+0000 2024-06-27T22:22:38.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: test_wait_for_scrub: wait_for_scrub 1.0 2024-06-27T22:22:31.006777+0000 2024-06-27T22:22:38.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-27T22:22:38.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-27T22:22:31.006777+0000 2024-06-27T22:22:38.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-27T22:22:38.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-27T22:22:38.457 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:22:38.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:22:38.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:22:38.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:22:38.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:22:38.457 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:22:38.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:22:31.006777+0000 '>' 2024-06-27T22:22:31.006777+0000 2024-06-27T22:22:38.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:22:39.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:22:39.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:22:39.879 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:22:39.879 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:22:39.879 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:22:39.879 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:22:39.880 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:22:40.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:22:31.006777+0000 '>' 2024-06-27T22:22:31.006777+0000 2024-06-27T22:22:40.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:22:41.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:22:41.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-06-27T22:22:41.309 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:22:41.309 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:22:41.309 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:22:41.310 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:22:41.310 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:22:41.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:22:38.806279+0000 '>' 2024-06-27T22:22:31.006777+0000 2024-06-27T22:22:41.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-06-27T22:22:41.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: test_wait_for_scrub: kill_daemons td/ceph-helpers KILL osd 2024-06-27T22:22:41.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:22:41.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:22:41.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:22:41.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:22:41.736 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:22:41.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:22:41.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-06-27T22:22:41.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:22:41.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:22:41.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:22:41.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:22:42.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: test_wait_for_scrub: last_scrub=2024-06-27T22:22:38.806279+0000 2024-06-27T22:22:42.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: test_wait_for_scrub: TIMEOUT=1 2024-06-27T22:22:42.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: test_wait_for_scrub: wait_for_scrub 1.0 2024-06-27T22:22:38.806279+0000 2024-06-27T22:22:42.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-06-27T22:22:42.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-06-27T22:22:38.806279+0000 2024-06-27T22:22:42.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-06-27T22:22:42.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-06-27T22:22:42.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-06-27T22:22:42.263 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-27T22:22:42.263 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-27T22:22:42.263 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-27T22:22:42.263 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-27T22:22:42.263 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-27T22:22:42.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-06-27T22:22:38.806279+0000 '>' 2024-06-27T22:22:38.806279+0000 2024-06-27T22:22:42.683 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-06-27T22:22:43.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-06-27T22:22:43.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-06-27T22:22:43.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: wait_for_scrub: return 1 2024-06-27T22:22:43.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: test_wait_for_scrub: teardown td/ceph-helpers 2024-06-27T22:22:43.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:22:43.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:22:43.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:22:43.687 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:22:43.687 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:22:43.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:22:43.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:22:43.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:22:43.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:22:43.797 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:22:43.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:22:43.798 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:22:43.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:22:43.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:22:43.799 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:22:43.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:22:43.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:22:43.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:22:43.801 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:22:43.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:22:43.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:22:43.803 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:22:43.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:22:43.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:22:43.810 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:22:43.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:22:43.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:22:43.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:22:43.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-06-27T22:22:43.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_osd_id_used_by_pgs td/ceph-helpers 2024-06-27T22:22:43.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1501: test_wait_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-06-27T22:22:43.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1503: test_wait_osd_id_used_by_pgs: setup td/ceph-helpers 2024-06-27T22:22:43.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-06-27T22:22:43.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-06-27T22:22:43.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:22:43.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:22:43.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:22:43.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:22:43.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:22:43.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:22:43.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:22:43.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:22:43.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:22:43.815 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:22:43.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:22:43.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:22:43.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:22:43.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:22:43.818 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:22:43.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:22:43.818 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:22:43.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:22:43.819 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:22:43.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:22:43.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:22:43.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:22:43.822 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:22:43.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:22:43.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:22:43.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:22:43.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:22:43.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:22:43.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-06-27T22:22:43.825 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:22:43.825 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:22:43.825 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:22:43.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70124 2024-06-27T22:22:43.827 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:22:43.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:22:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:22:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-06-27T22:22:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1504: test_wait_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:22:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-06-27T22:22:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:22:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:22:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:22:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-06-27T22:22:43.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:22:44.303 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:22:44.303 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:22:44.303 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:22:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:22:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:22:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:22:44.304 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:22:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-27T22:22:44.514 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:22:44.515 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:22:44.515 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:22:44.515 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:22:44.515 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:22:44.516 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:22:44.516 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:22:44.516 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:22:44.516 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:22:44.517 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:22:44.517 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:22:44.517 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:22:44.517 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:22:44.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:22:44.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get fsid 2024-06-27T22:22:44.632 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:22:44.632 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:22:44.632 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:22:44.633 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:22:44.633 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:22:44.633 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:22:44.633 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:22:44.633 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:22:44.634 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:22:44.634 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:22:44.634 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:22:44.635 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70124/ceph-mon.a.asok 2024-06-27T22:22:44.635 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:22:44.635 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70124/ceph-mon.a.asok config get mon_host 2024-06-27T22:22:44.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1505: test_wait_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-06-27T22:22:44.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-06-27T22:22:44.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:22:44.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:22:44.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:22:44.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-06-27T22:22:44.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:22:45.682 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:22:45.682 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:22:45.682 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:22:45.682 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:22:45.683 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:22:45.683 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:22:45.683 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:22:45.683 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:22:45.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:22:45.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1506: test_wait_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-06-27T22:22:45.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-06-27T22:22:45.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:22:45.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:22:45.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:22:45.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-06-27T22:22:45.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-06-27T22:22:45.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:22:45.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:22:45.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:22:45.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-06-27T22:22:45.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-06-27T22:22:45.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:22:45.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:22:45.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-06-27T22:22:45.756 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:22:45.756 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:22:45.756 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:22:45.757 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:22:45.758 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:22:45.758 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:22:45.758 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:22:45.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' 2024-06-27T22:22:45.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:22:45.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:22:45.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:22:45.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-06-27T22:22:45.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-06-27T22:22:45.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:22:45.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:22:45.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:22:45.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:22:45.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:22:45.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:22:45.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-06-27T22:22:45.759 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:22:45.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1b827a75-e9ca-43cd-9344-7460c5d2a89c 2024-06-27T22:22:45.760 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1b827a75-e9ca-43cd-9344-7460c5d2a89c' 2024-06-27T22:22:45.760 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 1b827a75-e9ca-43cd-9344-7460c5d2a89c 2024-06-27T22:22:45.760 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:22:45.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA15n1mCHiBLhAAh2bDxYNKjUHsJTGNWaT70A== 2024-06-27T22:22:45.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA15n1mCHiBLhAAh2bDxYNKjUHsJTGNWaT70A=="}' 2024-06-27T22:22:45.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1b827a75-e9ca-43cd-9344-7460c5d2a89c -i td/ceph-helpers/0/new.json 2024-06-27T22:22:46.101 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:22:46.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-06-27T22:22:46.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA15n1mCHiBLhAAh2bDxYNKjUHsJTGNWaT70A== --osd-uuid 1b827a75-e9ca-43cd-9344-7460c5d2a89c 2024-06-27T22:22:46.148 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:46.148+0000 7fcb62fd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:22:46.179 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:46.180+0000 7fcb62fd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:22:46.180 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:46.180+0000 7fcb62fd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:22:46.181 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:46.180+0000 7fcb62fd17c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:22:46.181 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:46.180+0000 7fcb62fd17c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:22:46.181 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:46.180+0000 7fcb62fd17c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-06-27T22:22:46.209 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:46.212+0000 7fcb62fd17c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-06-27T22:22:52.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-06-27T22:22:52.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:22:52.756 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:22:52.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:22:52.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:22:53.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:22:53.378 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:22:53.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:22:53.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:22:53.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=0e761281-cb1c-448b-bd43-8b5efce19754 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.70124/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:22:53.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:22:53.412 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:53.411+0000 7f2d725fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:22:53.426 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:53.427+0000 7f2d725fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:22:53.427 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:53.427+0000 7f2d725fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:22:53.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:22:53.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:22:53.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:22:53.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:22:53.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:22:53.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:53.792 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:22:53.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:22:53.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:53.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:54.003 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:54.003+0000 7f2d725fc7c0 -1 Falling back to public interface 2024-06-27T22:22:54.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:55.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:55.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:55.236 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:22:55.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:22:55.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:55.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:55.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:56.659 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:22:56.659+0000 7f2d725fc7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:22:56.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:56.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:56.673 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:22:56.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:22:56.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:56.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:57.275 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:58.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:58.277 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:58.277 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:22:58.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:22:58.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:58.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:22:58.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:22:59.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:22:59.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:22:59.761 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:22:59.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:22:59.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:22:59.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:00.238 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:01.239 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:01.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:01.240 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:23:01.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:23:01.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:01.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:01.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:02.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:02.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:02.728 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:23:02.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:23:02.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:02.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:03.226 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:04.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:04.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:04.228 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:23:04.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:23:04.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:04.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:04.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:05.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:05.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:05.737 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:23:05.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:23:05.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:05.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:06.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:07.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:07.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:07.185 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:23:07.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:23:07.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:07.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:07.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:08.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:08.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:08.849 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:23:08.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:23:08.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:08.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:09.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:10.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:10.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:10.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:23:10.463 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:23:10.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:10.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:11.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:12.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:12.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:12.597 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:23:12.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:23:12.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:12.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:13.109 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:14.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:14.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:14.111 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:23:14.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:23:14.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:14.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:14.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:15.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:15.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:15.851 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:23:15.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:23:15.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:15.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:16.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:17.629 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:17.629 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:17.629 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:23:17.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:23:17.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:17.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:18.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:19.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:19.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:19.466 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:23:19.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:23:19.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:19.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:20.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:21.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:21.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:21.535 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:23:21.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:23:21.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:21.535 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:22.023 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/188193500,v1:127.0.0.1:6803/188193500] [v2:127.0.0.1:6804/188193500,v1:127.0.0.1:6805/188193500] exists,up 1b827a75-e9ca-43cd-9344-7460c5d2a89c 2024-06-27T22:23:22.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:23:22.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:23:22.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:23:22.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1507: test_wait_osd_id_used_by_pgs: create_rbd_pool 2024-06-27T22:23:22.024 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-27T22:23:22.481 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' does not exist 2024-06-27T22:23:22.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-27T22:23:22.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-27T22:23:23.194 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'rbd' already exists 2024-06-27T22:23:23.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:23:24.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-27T22:23:25.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1508: test_wait_osd_id_used_by_pgs: wait_for_clean 2024-06-27T22:23:25.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:23:25.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:23:25.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:23:25.470 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:23:25.470 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:23:25.470 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:23:25.471 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:23:25.471 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:23:25.471 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:23:25.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:23:25.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:23:25.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:23:25.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:23:25.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:23:25.599 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:23:26.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-06-27T22:23:26.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:23:26.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:23:26.045 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:23:26.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836488 2024-06-27T22:23:26.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836488 2024-06-27T22:23:26.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836488' 2024-06-27T22:23:26.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:23:26.208 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836488 2024-06-27T22:23:26.208 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:23:26.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:23:26.209 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836488 2024-06-27T22:23:26.209 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:23:26.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836488 2024-06-27T22:23:26.210 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836488 2024-06-27T22:23:26.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-06-27T22:23:26.211 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:23:26.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836486 -lt 21474836488 2024-06-27T22:23:26.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:23:27.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:23:27.636 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:23:28.072 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836488 -lt 21474836488 2024-06-27T22:23:28.073 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:23:28.073 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:23:28.073 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:23:28.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-27T22:23:28.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:23:28.667 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:23:28.667 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:23:28.667 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:23:28.667 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:23:28.667 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:23:28.667 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:23:29.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-27T22:23:29.169 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:23:29.169 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:23:29.169 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:23:29.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-27T22:23:29.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:23:29.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:23:29.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1509: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 0 8 2024-06-27T22:23:29.734 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=0 2024-06-27T22:23:29.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=8 2024-06-27T22:23:29.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-06-27T22:23:29.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-06-27T22:23:29.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 300 / 5 )) 2024-06-27T22:23:29.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1489: wait_osd_id_used_by_pgs: echo 0 2024-06-27T22:23:29.735 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:23:29.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-06-27T22:23:29.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: grep -c 0 2024-06-27T22:23:29.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-06-27T22:23:29.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-06-27T22:23:29.736 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-06-27T22:23:30.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: test 8 = 8 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1493: wait_osd_id_used_by_pgs: status=0 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: wait_osd_id_used_by_pgs: break 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 0 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: TIMEOUT=1 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 123 5 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=123 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=5 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 1 / 5 )) 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 1 2024-06-27T22:23:30.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1511: test_wait_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-06-27T22:23:30.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-06-27T22:23:30.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:23:30.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-06-27T22:23:30.174 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:23:30.174 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:23:30.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:23:30.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:23:30.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:23:30.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:23:30.278 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:23:30.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:23:30.279 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:23:30.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:23:30.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:23:30.280 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:23:30.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:23:30.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:23:30.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:23:30.282 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:23:30.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:23:30.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:23:30.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-06-27T22:23:30.290 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:23:30.290 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:23:30.290 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70124 2024-06-27T22:23:30.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70124 2024-06-27T22:23:30.290 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:23:30.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:23:30.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: : exit 0 2024-06-27T22:23:30.292 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-27T22:23:30.292 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-27T22:23:30.302 INFO:tasks.workunit:Running workunit misc/test-snaptrim-stats.sh... 2024-06-27T22:23:30.302 DEBUG:teuthology.orchestra.run.smithi042:workunit test misc/test-snaptrim-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh 2024-06-27T22:23:30.355 INFO:tasks.workunit.client.0.smithi042.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-27T22:23:30.360 INFO:tasks.workunit.client.0.smithi042.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-27T22:23:30.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T22:23:30.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T22:23:30.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-06-27T22:23:30.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-06-27T22:23:30.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-06-27T22:23:30.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-06-27T22:23:30.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-06-27T22:23:30.360 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-06-27T22:23:30.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/test-snaptrim-stats 2024-06-27T22:23:30.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:21: run: local dir=td/test-snaptrim-stats 2024-06-27T22:23:30.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:22: run: shift 2024-06-27T22:23:30.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-06-27T22:23:30.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-06-27T22:23:30.361 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:25: run: export CEPH_ARGS 2024-06-27T22:23:30.361 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: uuidgen 2024-06-27T22:23:30.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: CEPH_ARGS+='--fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none ' 2024-06-27T22:23:30.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-06-27T22:23:30.362 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:28: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-06-27T22:23:30.362 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: set 2024-06-27T22:23:30.362 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-27T22:23:30.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: local 'funcs=TEST_snaptrim_stats 2024-06-27T22:23:30.364 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_snaptrim_stats_multiple_snaps' 2024-06-27T22:23:30.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-06-27T22:23:30.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-06-27T22:23:30.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-06-27T22:23:30.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-06-27T22:23:30.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-27T22:23:30.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:23:30.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-27T22:23:30.364 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:23:30.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:23:30.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:23:30.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:23:30.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:23:30.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:23:30.366 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:23:30.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:23:30.366 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:23:30.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:23:30.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:23:30.367 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:23:30.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:23:30.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:23:30.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:23:30.368 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:23:30.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:23:30.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:23:30.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-27T22:23:30.371 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:23:30.371 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:23:30.371 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:23:30.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168036 2024-06-27T22:23:30.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:23:30.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:23:30.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-06-27T22:23:30.373 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:23:30.373 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:23:30.373 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:23:30.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.168036 2024-06-27T22:23:30.375 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats td/test-snaptrim-stats 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:39: TEST_snaptrim_stats: local dir=td/test-snaptrim-stats 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:40: TEST_snaptrim_stats: local poolname=test 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:41: TEST_snaptrim_stats: local OSDS=3 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:42: TEST_snaptrim_stats: local PGNUM=8 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:43: TEST_snaptrim_stats: local PGPNUM=8 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:44: TEST_snaptrim_stats: local objects=10 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:45: TEST_snaptrim_stats: local WAIT_FOR_UPDATE=10 2024-06-27T22:23:30.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:47: TEST_snaptrim_stats: setup td/test-snaptrim-stats 2024-06-27T22:23:30.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-06-27T22:23:30.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-06-27T22:23:30.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-27T22:23:30.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:23:30.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-27T22:23:30.377 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:23:30.377 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:23:30.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:23:30.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:23:30.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:23:30.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:23:30.378 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:23:30.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:23:30.378 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:23:30.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:23:30.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:23:30.378 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:23:30.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:23:30.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:23:30.379 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:23:30.379 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:23:30.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:23:30.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:23:30.381 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-27T22:23:30.382 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:23:30.382 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:23:30.382 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:23:30.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168036 2024-06-27T22:23:30.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:23:30.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:23:30.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-06-27T22:23:30.384 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:23:30.384 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:23:30.384 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:23:30.384 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.168036 2024-06-27T22:23:30.386 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:23:30.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:23:30.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:23:30.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-06-27T22:23:30.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:48: TEST_snaptrim_stats: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-06-27T22:23:30.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-06-27T22:23:30.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:23:30.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:23:30.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:23:30.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-06-27T22:23:30.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-06-27T22:23:30.867 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:23:30.867 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:23:30.867 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:23:30.867 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:23:30.867 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:23:30.868 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:23:30.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:23:30.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-06-27T22:23:31.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:23:31.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:23:31.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:23:31.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:23:31.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:23:31.081 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:23:31.082 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:23:31.082 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:23:31.082 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:23:31.082 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:23:31.082 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:23:31.082 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:23:31.082 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.168036/ceph-mon.a.asok 2024-06-27T22:23:31.083 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:23:31.083 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.168036/ceph-mon.a.asok config get fsid 2024-06-27T22:23:31.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:23:31.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:23:31.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:23:31.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:23:31.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:23:31.198 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:23:31.198 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:23:31.198 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:23:31.199 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:23:31.199 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:23:31.199 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:23:31.200 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.168036/ceph-mon.a.asok 2024-06-27T22:23:31.200 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:23:31.200 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.168036/ceph-mon.a.asok config get mon_host 2024-06-27T22:23:31.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:49: TEST_snaptrim_stats: run_mgr td/test-snaptrim-stats x 2024-06-27T22:23:31.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-06-27T22:23:31.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:23:31.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:23:31.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:23:31.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-06-27T22:23:31.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:23:32.226 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:23:32.227 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:23:32.227 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:23:32.227 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:23:32.227 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:23:32.227 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:23:32.227 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:23:32.228 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:23:32.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:23:32.263 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: expr 3 - 1 2024-06-27T22:23:32.264 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: seq 0 2 2024-06-27T22:23:32.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-27T22:23:32.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:23:32.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-27T22:23:32.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:23:32.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:23:32.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:23:32.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-06-27T22:23:32.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-27T22:23:32.268 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:23:32.269 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:23:32.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-06-27T22:23:32.271 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:23:32.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b6b80832-6da3-4fe7-9615-4cf3bfae574d 2024-06-27T22:23:32.271 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b6b80832-6da3-4fe7-9615-4cf3bfae574d' 2024-06-27T22:23:32.271 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 b6b80832-6da3-4fe7-9615-4cf3bfae574d 2024-06-27T22:23:32.272 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:23:32.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBk5n1mXFMyERAAJCQN94xJhmz4p6DywmcaYw== 2024-06-27T22:23:32.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBk5n1mXFMyERAAJCQN94xJhmz4p6DywmcaYw=="}' 2024-06-27T22:23:32.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b6b80832-6da3-4fe7-9615-4cf3bfae574d -i td/test-snaptrim-stats/0/new.json 2024-06-27T22:23:32.615 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:23:32.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-06-27T22:23:32.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBk5n1mXFMyERAAJCQN94xJhmz4p6DywmcaYw== --osd-uuid b6b80832-6da3-4fe7-9615-4cf3bfae574d 2024-06-27T22:23:32.667 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:32.667+0000 7f4717a407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:23:32.683 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:32.683+0000 7f4717a407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:23:32.684 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:32.683+0000 7f4717a407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:23:32.685 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:32.683+0000 7f4717a407c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-27T22:23:32.685 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:32.683+0000 7f4717a407c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-27T22:23:32.685 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:32.683+0000 7f4717a407c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-27T22:23:32.732 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:32.731+0000 7f4717a407c0 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-06-27T22:23:39.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-06-27T22:23:39.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:23:39.296 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:23:39.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:23:39.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:23:39.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:23:39.914 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:23:39.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:23:39.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:23:39.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:23:39.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:23:39.945 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:39.943+0000 7f156934a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:23:39.974 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:39.975+0000 7f156934a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:23:39.976 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:39.975+0000 7f156934a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:23:40.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:23:40.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:23:40.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:23:40.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:23:40.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:23:40.376 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:23:40.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:40.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:23:40.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:40.377 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:40.555 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:40.555+0000 7f156934a7c0 -1 Falling back to public interface 2024-06-27T22:23:40.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:41.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:41.865 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:41.865 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:23:41.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:23:41.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:41.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:42.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:43.246 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:23:43.247+0000 7f156934a7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:23:43.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:43.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:43.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:23:43.340 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:23:43.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:43.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:43.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:44.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:44.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:44.991 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:23:44.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:23:44.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:44.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:45.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:46.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:46.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:46.543 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:23:46.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:23:46.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:46.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:47.132 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:48.134 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:48.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:48.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:23:48.135 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:23:48.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:48.135 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:48.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:49.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:49.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:49.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:23:49.627 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:23:49.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:49.627 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:50.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:51.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:51.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:51.178 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:23:51.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:23:51.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:51.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:51.623 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:52.624 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:52.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:52.625 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:23:52.625 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:23:52.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:52.626 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:53.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:54.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:54.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:54.146 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:23:54.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:23:54.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:54.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:54.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:55.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:55.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:55.610 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:23:55.610 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:23:55.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:55.611 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:56.103 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:57.104 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:57.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:57.105 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:23:57.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:23:57.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:57.105 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:58.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:23:59.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:23:59.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:23:59.035 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:23:59.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:23:59.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:23:59.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:23:59.525 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:00.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:00.527 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:00.527 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:24:00.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:24:00.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:00.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:24:01.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:02.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:02.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:02.618 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:24:02.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:24:02.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:02.619 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:24:03.593 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:04.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:04.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:04.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:24:04.595 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:24:04.595 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:04.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:24:05.504 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:06.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:06.506 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:06.506 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:24:06.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:24:06.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:06.507 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:24:07.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:08.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:08.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:08.382 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:24:08.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:24:08.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:08.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:24:08.854 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1654856642,v1:127.0.0.1:6803/1654856642] [v2:127.0.0.1:6804/1654856642,v1:127.0.0.1:6805/1654856642] exists,up b6b80832-6da3-4fe7-9615-4cf3bfae574d 2024-06-27T22:24:08.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:24:08.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:24:08.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:24:08.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-27T22:24:08.858 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:24:08.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-27T22:24:08.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:24:08.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:24:08.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:24:08.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-06-27T22:24:08.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-27T22:24:08.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:24:08.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:24:08.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:24:08.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-06-27T22:24:08.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:24:08.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-06-27T22:24:08.861 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:24:08.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=81cecf7c-7e53-4919-b5c4-908adcc6a0d8 2024-06-27T22:24:08.862 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 81cecf7c-7e53-4919-b5c4-908adcc6a0d8 2024-06-27T22:24:08.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 81cecf7c-7e53-4919-b5c4-908adcc6a0d8' 2024-06-27T22:24:08.863 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:24:08.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCI5n1mMfdXNBAAgahRSa7FjUX5mkMe+m73tw== 2024-06-27T22:24:08.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCI5n1mMfdXNBAAgahRSa7FjUX5mkMe+m73tw=="}' 2024-06-27T22:24:08.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 81cecf7c-7e53-4919-b5c4-908adcc6a0d8 -i td/test-snaptrim-stats/1/new.json 2024-06-27T22:24:09.451 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:24:09.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-06-27T22:24:09.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQCI5n1mMfdXNBAAgahRSa7FjUX5mkMe+m73tw== --osd-uuid 81cecf7c-7e53-4919-b5c4-908adcc6a0d8 2024-06-27T22:24:09.505 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:09.503+0000 7fecdb8177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:09.507 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:09.507+0000 7fecdb8177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:09.508 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:09.507+0000 7fecdb8177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:09.509 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:09.507+0000 7fecdb8177c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-27T22:24:09.509 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:09.507+0000 7fecdb8177c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-27T22:24:09.509 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:09.507+0000 7fecdb8177c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-27T22:24:09.585 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:09.583+0000 7fecdb8177c0 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-06-27T22:24:15.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-06-27T22:24:15.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:24:15.710 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:24:15.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:24:15.711 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:24:16.299 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:24:16.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:24:16.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:24:16.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:24:16.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:24:16.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:24:16.332 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:16.331+0000 7ff66f1077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:16.341 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:16.339+0000 7ff66f1077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:16.342 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:16.343+0000 7ff66f1077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:16.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:24:16.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:24:16.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:24:16.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:24:16.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:24:16.773 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:16.773 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:24:16.774 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:24:16.774 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:16.774 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:16.915 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:16.915+0000 7ff66f1077c0 -1 Falling back to public interface 2024-06-27T22:24:17.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:18.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:18.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:18.221 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:24:18.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:24:18.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:18.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:18.714 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:19.631 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:19.631+0000 7ff66f1077c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:24:19.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:19.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:19.716 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:24:19.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:24:19.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:19.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:20.385 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:21.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:21.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:21.387 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:24:21.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:24:21.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:21.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:21.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:22.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:22.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:22.973 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:24:22.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:24:22.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:22.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:23.594 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:24.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:24.596 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:24.597 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:24:24.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:24:24.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:24.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:25.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:26.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:26.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:26.049 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:24:26.050 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:24:26.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:26.050 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:26.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:27.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:27.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:27.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:24:27.639 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:24:27.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:27.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:28.137 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:29.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:29.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:29.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:24:29.140 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:24:29.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:29.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:29.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:30.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:30.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:30.764 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:24:30.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:24:30.765 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:30.765 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:31.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:32.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:32.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:32.217 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:24:32.217 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:24:32.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:32.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:32.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:33.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:33.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:33.832 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:24:33.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:24:33.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:33.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:34.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:35.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:35.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:35.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:24:35.556 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:24:35.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:35.557 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:36.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:37.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:37.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:37.412 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:24:37.413 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:24:37.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:37.413 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:37.918 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:38.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:38.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:38.921 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:24:38.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:24:38.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:38.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:39.682 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:40.684 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:40.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:40.685 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:24:40.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:24:40.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:40.686 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:41.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:42.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:42.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:42.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:24:42.511 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:24:42.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:42.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:43.490 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:44.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:44.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:44.493 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:24:44.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:24:44.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:44.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:45.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:46.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:46.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:46.224 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:24:46.225 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:24:46.225 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:46.225 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:24:46.651 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/376770709,v1:127.0.0.1:6811/376770709] [v2:127.0.0.1:6812/376770709,v1:127.0.0.1:6813/376770709] exists,up 81cecf7c-7e53-4919-b5c4-908adcc6a0d8 2024-06-27T22:24:46.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:24:46.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:24:46.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:24:46.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-27T22:24:46.654 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:24:46.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:24:46.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-27T22:24:46.656 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:24:46.656 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:24:46.656 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:24:46.656 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:24:46.656 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:24:46.656 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:24:46.656 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:24:46.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:24:46.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:24:46.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:24:46.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:24:46.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-27T22:24:46.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-27T22:24:46.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:24:46.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:24:46.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:24:46.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:24:46.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:24:46.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:24:46.657 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-06-27T22:24:46.658 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:24:46.658 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6e141092-d5c6-4006-9a60-4aa450ebc72e 2024-06-27T22:24:46.658 INFO:tasks.workunit.client.0.smithi042.stdout:add osd2 6e141092-d5c6-4006-9a60-4aa450ebc72e 2024-06-27T22:24:46.659 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6e141092-d5c6-4006-9a60-4aa450ebc72e' 2024-06-27T22:24:46.659 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:24:46.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCu5n1m5Xo/KBAA9I04fD8IQ0nuK1wd4DV+bw== 2024-06-27T22:24:46.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCu5n1m5Xo/KBAA9I04fD8IQ0nuK1wd4DV+bw=="}' 2024-06-27T22:24:46.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6e141092-d5c6-4006-9a60-4aa450ebc72e -i td/test-snaptrim-stats/2/new.json 2024-06-27T22:24:47.213 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:24:47.228 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-06-27T22:24:47.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQCu5n1m5Xo/KBAA9I04fD8IQ0nuK1wd4DV+bw== --osd-uuid 6e141092-d5c6-4006-9a60-4aa450ebc72e 2024-06-27T22:24:47.263 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:47.263+0000 7fdeb82437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:47.265 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:47.263+0000 7fdeb82437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:47.266 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:47.267+0000 7fdeb82437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:47.267 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:47.267+0000 7fdeb82437c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-27T22:24:47.267 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:47.267+0000 7fdeb82437c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-27T22:24:47.267 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:47.267+0000 7fdeb82437c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-27T22:24:47.304 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:47.303+0000 7fdeb82437c0 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-06-27T22:24:54.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-06-27T22:24:54.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:24:54.083 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd2 key to auth repository 2024-06-27T22:24:54.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-27T22:24:54.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:24:54.668 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T22:24:54.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-27T22:24:54.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:24:54.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:24:54.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:24:54.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:24:54.700 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:54.699+0000 7fc8df7ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:54.724 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:54.723+0000 7fc8df7ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:54.726 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:54.727+0000 7fc8df7ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:24:55.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-27T22:24:55.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:24:55.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T22:24:55.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:24:55.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:24:55.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:55.130 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:24:55.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:24:55.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:55.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:24:55.307 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:55.307+0000 7fc8df7ca7c0 -1 Falling back to public interface 2024-06-27T22:24:55.590 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:56.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:56.591 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:56.592 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:24:56.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:24:56.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:56.592 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:24:57.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:57.987 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:24:57.987+0000 7fc8df7ca7c0 -1 osd.2 0 log_to_monitors true 2024-06-27T22:24:58.063 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:58.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:58.064 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:24:58.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:24:58.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:58.065 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:24:58.784 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:24:59.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:24:59.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:24:59.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:24:59.786 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:24:59.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:24:59.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:00.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:01.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:01.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:01.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:25:01.234 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:25:01.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:01.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:01.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:02.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:02.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:02.808 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:25:02.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:25:02.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:02.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:03.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:04.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:04.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:04.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:25:04.296 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:25:04.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:04.296 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:04.889 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:05.890 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:05.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:05.891 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:25:05.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:25:05.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:05.892 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:06.340 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:07.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:07.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:07.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:25:07.342 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:25:07.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:07.343 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:07.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:08.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:08.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:08.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:25:08.850 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:25:08.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:08.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:09.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:10.309 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:10.309 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:10.309 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:25:10.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:25:10.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:10.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:10.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:11.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:11.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:11.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:25:11.807 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:25:11.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:11.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:12.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:13.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:13.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:13.587 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:25:13.587 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:25:13.587 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:13.587 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:14.614 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:15.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:15.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:15.617 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:25:15.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:25:15.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:15.617 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:16.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:17.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:17.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:17.763 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:25:17.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:25:17.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:17.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:18.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:19.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:19.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:19.886 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:25:19.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:25:19.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:19.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:21.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:22.309 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:22.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:22.310 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:25:22.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:25:22.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:22.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:23.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:25:24.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:25:24.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:25:24.145 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:25:24.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:25:24.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:25:24.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:25:24.706 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2762079215,v1:127.0.0.1:6819/2762079215] [v2:127.0.0.1:6820/2762079215,v1:127.0.0.1:6821/2762079215] exists,up 6e141092-d5c6-4006-9a60-4aa450ebc72e 2024-06-27T22:25:24.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:25:24.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:25:24.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:25:24.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:56: TEST_snaptrim_stats: ceph osd set noscrub 2024-06-27T22:25:25.487 INFO:tasks.workunit.client.0.smithi042.stderr:noscrub is set 2024-06-27T22:25:25.512 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:57: TEST_snaptrim_stats: ceph osd set nodeep-scrub 2024-06-27T22:25:26.350 INFO:tasks.workunit.client.0.smithi042.stderr:nodeep-scrub is set 2024-06-27T22:25:26.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:60: TEST_snaptrim_stats: create_pool test 8 8 2024-06-27T22:25:26.373 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-06-27T22:25:27.013 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'test' already exists 2024-06-27T22:25:27.029 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:25:28.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:61: TEST_snaptrim_stats: wait_for_clean 2024-06-27T22:25:28.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:25:28.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:25:28.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:25:28.032 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:25:28.033 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:25:28.033 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:25:28.034 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:25:28.034 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:25:28.034 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:25:28.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:25:28.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:25:28.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:25:28.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:25:28.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:25:28.160 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:25:28.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T22:25:28.827 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T22:25:28.827 INFO:tasks.workunit.client.0.smithi042.stderr:2' 2024-06-27T22:25:28.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:25:28.828 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:25:28.828 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:25:29.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836504 2024-06-27T22:25:29.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836504 2024-06-27T22:25:29.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504' 2024-06-27T22:25:29.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:25:29.067 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T22:25:29.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672974 2024-06-27T22:25:29.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672974 2024-06-27T22:25:29.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672974' 2024-06-27T22:25:29.328 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:25:29.329 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T22:25:29.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509453 2024-06-27T22:25:29.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509453 2024-06-27T22:25:29.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672974 2-64424509453' 2024-06-27T22:25:29.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:25:29.492 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836504 2024-06-27T22:25:29.492 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:25:29.493 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:25:29.494 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836504 2024-06-27T22:25:29.494 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:25:29.495 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836504 2024-06-27T22:25:29.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836504 2024-06-27T22:25:29.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-06-27T22:25:29.495 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:25:29.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836504 2024-06-27T22:25:29.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:25:29.925 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:25:29.925 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672974 2024-06-27T22:25:29.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T22:25:29.927 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672974 2024-06-27T22:25:29.927 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:25:29.929 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672974 2024-06-27T22:25:29.929 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672974 2024-06-27T22:25:29.929 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-06-27T22:25:29.929 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T22:25:30.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672972 -lt 42949672974 2024-06-27T22:25:30.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:25:31.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:25:31.381 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T22:25:31.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672975 -lt 42949672974 2024-06-27T22:25:31.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:25:31.864 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509453 2024-06-27T22:25:31.864 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:25:31.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T22:25:31.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:25:31.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509453 2024-06-27T22:25:31.868 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509453 2024-06-27T22:25:31.868 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509453 2024-06-27T22:25:31.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-27T22:25:31.869 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T22:25:32.347 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509453 2024-06-27T22:25:32.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:25:32.349 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:25:32.349 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:25:32.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-06-27T22:25:32.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:25:32.988 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:25:32.988 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:25:32.988 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:25:32.988 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:25:32.988 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:25:32.989 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:25:33.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-27T22:25:33.422 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:25:33.422 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:25:33.422 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:25:33.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-06-27T22:25:33.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:25:33.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:25:33.991 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: ceph osd dump 2024-06-27T22:25:33.991 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: awk '{ print $2 }' 2024-06-27T22:25:33.991 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: grep '^pool.*['\'']test['\'']' 2024-06-27T22:25:34.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: poolid=1 2024-06-27T22:25:34.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:65: TEST_snaptrim_stats: TESTDATA=testdata.1 2024-06-27T22:25:34.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:66: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-06-27T22:25:34.460 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records in 2024-06-27T22:25:34.461 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records out 2024-06-27T22:25:34.461 INFO:tasks.workunit.client.0.smithi042.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000126497 s, 32.4 MB/s 2024-06-27T22:25:34.461 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: seq 1 10 2024-06-27T22:25:34.462 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:34.462 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj1 testdata.1 2024-06-27T22:25:34.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:34.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj2 testdata.1 2024-06-27T22:25:34.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:34.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj3 testdata.1 2024-06-27T22:25:34.997 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:34.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj4 testdata.1 2024-06-27T22:25:35.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:35.329 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj5 testdata.1 2024-06-27T22:25:35.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:35.509 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj6 testdata.1 2024-06-27T22:25:35.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:35.703 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj7 testdata.1 2024-06-27T22:25:35.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:35.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj8 testdata.1 2024-06-27T22:25:36.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:36.039 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj9 testdata.1 2024-06-27T22:25:36.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:36.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj10 testdata.1 2024-06-27T22:25:36.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:71: TEST_snaptrim_stats: rm -f testdata.1 2024-06-27T22:25:36.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:74: TEST_snaptrim_stats: SNAP=1 2024-06-27T22:25:36.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:75: TEST_snaptrim_stats: rados -p test mksnap snap1 2024-06-27T22:25:36.558 INFO:tasks.workunit.client.0.smithi042.stdout:created pool test snap snap1 2024-06-27T22:25:36.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:76: TEST_snaptrim_stats: TESTDATA=testdata.2 2024-06-27T22:25:36.562 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:77: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-06-27T22:25:36.563 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records in 2024-06-27T22:25:36.563 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records out 2024-06-27T22:25:36.563 INFO:tasks.workunit.client.0.smithi042.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000141799 s, 28.9 MB/s 2024-06-27T22:25:36.564 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: seq 1 10 2024-06-27T22:25:36.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:36.566 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj1 testdata.2 2024-06-27T22:25:36.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:36.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj2 testdata.2 2024-06-27T22:25:37.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:37.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj3 testdata.2 2024-06-27T22:25:37.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:37.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj4 testdata.2 2024-06-27T22:25:37.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:37.432 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj5 testdata.2 2024-06-27T22:25:37.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:37.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj6 testdata.2 2024-06-27T22:25:37.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:37.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj7 testdata.2 2024-06-27T22:25:37.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:37.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj8 testdata.2 2024-06-27T22:25:38.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:38.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj9 testdata.2 2024-06-27T22:25:38.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-06-27T22:25:38.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj10 testdata.2 2024-06-27T22:25:38.460 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:82: TEST_snaptrim_stats: rm -f testdata.2 2024-06-27T22:25:38.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:85: TEST_snaptrim_stats: rados -p test rmsnap snap1 2024-06-27T22:25:38.636 INFO:tasks.workunit.client.0.smithi042.stdout:removed pool test snap snap1 2024-06-27T22:25:38.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:88: TEST_snaptrim_stats: wait_for_clean 2024-06-27T22:25:38.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:25:38.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:25:38.642 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:25:38.642 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:25:38.643 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:25:38.643 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:25:38.644 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:25:38.644 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:25:38.644 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:25:38.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:25:38.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:25:38.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:25:38.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:25:38.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:25:38.769 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:25:39.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T22:25:39.219 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T22:25:39.219 INFO:tasks.workunit.client.0.smithi042.stderr:2' 2024-06-27T22:25:39.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:25:39.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:25:39.219 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:25:39.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836508 2024-06-27T22:25:39.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836508 2024-06-27T22:25:39.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508' 2024-06-27T22:25:39.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:25:39.540 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T22:25:39.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-06-27T22:25:39.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-06-27T22:25:39.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672978' 2024-06-27T22:25:39.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:25:39.744 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T22:25:39.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509458 2024-06-27T22:25:39.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509458 2024-06-27T22:25:39.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672978 2-64424509458' 2024-06-27T22:25:39.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:25:39.906 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836508 2024-06-27T22:25:39.906 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:25:39.907 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:25:39.907 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-06-27T22:25:39.907 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:25:39.908 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836508 2024-06-27T22:25:39.908 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836508 2024-06-27T22:25:39.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-06-27T22:25:39.909 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:25:40.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836508 2024-06-27T22:25:40.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:25:41.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:25:41.395 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:25:41.839 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836508 2024-06-27T22:25:41.839 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:25:41.839 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-06-27T22:25:41.840 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:25:41.842 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T22:25:41.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-06-27T22:25:41.843 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:25:41.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-06-27T22:25:41.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-06-27T22:25:41.845 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672978 2024-06-27T22:25:41.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T22:25:42.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672978 2024-06-27T22:25:42.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:25:42.367 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509458 2024-06-27T22:25:42.367 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:25:42.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T22:25:42.370 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-06-27T22:25:42.370 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:25:42.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509458 2024-06-27T22:25:42.374 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509458 2024-06-27T22:25:42.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-06-27T22:25:42.374 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T22:25:42.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509458 2024-06-27T22:25:42.849 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:25:42.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:25:42.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:25:43.490 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-06-27T22:25:43.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:25:43.491 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:25:43.491 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:25:43.492 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:25:43.492 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:25:43.492 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:25:43.492 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:25:43.904 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-27T22:25:43.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:25:43.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:25:43.905 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:25:44.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-06-27T22:25:44.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:25:44.463 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:25:44.464 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:89: TEST_snaptrim_stats: sleep 10 2024-06-27T22:25:54.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:90: TEST_snaptrim_stats: local objects_trimmed=0 2024-06-27T22:25:54.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:91: TEST_snaptrim_stats: local snaptrim_duration_total=0.0 2024-06-27T22:25:54.467 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: expr 8 - 1 2024-06-27T22:25:54.469 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: seq 0 7 2024-06-27T22:25:54.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:25:54.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.0 2024-06-27T22:25:54.472 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.0 query 2024-06-27T22:25:54.472 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-27T22:25:54.644 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 0 + 2 2024-06-27T22:25:54.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=2 2024-06-27T22:25:54.646 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-27T22:25:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.0 query 2024-06-27T22:25:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-27T22:25:54.806 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 0.0 + 0.967458984 2024-06-27T22:25:54.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.967458984 2024-06-27T22:25:54.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:25:54.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.1 2024-06-27T22:25:54.808 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.1 query 2024-06-27T22:25:54.808 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-27T22:25:54.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 2 + 2 2024-06-27T22:25:54.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-06-27T22:25:54.967 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-27T22:25:54.968 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.1 query 2024-06-27T22:25:54.968 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-27T22:25:55.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .967458984 + 0.770958112 2024-06-27T22:25:55.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.738417096 2024-06-27T22:25:55.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:25:55.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.2 2024-06-27T22:25:55.121 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-27T22:25:55.122 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.2 query 2024-06-27T22:25:55.277 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 0 2024-06-27T22:25:55.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-06-27T22:25:55.278 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-27T22:25:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.2 query 2024-06-27T22:25:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-27T22:25:55.449 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.738417096 + 0.000566696 2024-06-27T22:25:55.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.738983792 2024-06-27T22:25:55.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:25:55.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.3 2024-06-27T22:25:55.451 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.3 query 2024-06-27T22:25:55.451 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-27T22:25:55.604 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 2 2024-06-27T22:25:55.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=6 2024-06-27T22:25:55.606 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-27T22:25:55.607 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.3 query 2024-06-27T22:25:55.607 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-27T22:25:55.765 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.738983792 + 0.571076713 2024-06-27T22:25:55.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.310060505 2024-06-27T22:25:55.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:25:55.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.4 2024-06-27T22:25:55.767 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.4 query 2024-06-27T22:25:55.767 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-27T22:25:55.924 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 6 + 1 2024-06-27T22:25:55.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=7 2024-06-27T22:25:55.925 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-27T22:25:55.926 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.4 query 2024-06-27T22:25:55.927 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-27T22:25:56.080 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.310060505 + 0.23921912 2024-06-27T22:25:56.080 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.549279625 2024-06-27T22:25:56.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:25:56.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.5 2024-06-27T22:25:56.082 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.5 query 2024-06-27T22:25:56.082 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-27T22:25:56.240 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 7 + 1 2024-06-27T22:25:56.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-06-27T22:25:56.242 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-27T22:25:56.243 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.5 query 2024-06-27T22:25:56.243 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-27T22:25:56.400 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.549279625 + 0.670901587 2024-06-27T22:25:56.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=3.220181212 2024-06-27T22:25:56.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:25:56.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.6 2024-06-27T22:25:56.402 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.6 query 2024-06-27T22:25:56.402 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-27T22:25:56.555 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 0 2024-06-27T22:25:56.556 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-06-27T22:25:56.557 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-27T22:25:56.558 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.6 query 2024-06-27T22:25:56.558 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-27T22:25:56.713 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 3.220181212 + 0.486824734 2024-06-27T22:25:56.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=3.707005946 2024-06-27T22:25:56.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:25:56.714 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.7 2024-06-27T22:25:56.714 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.7 query 2024-06-27T22:25:56.715 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-06-27T22:25:56.876 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 2 2024-06-27T22:25:56.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=10 2024-06-27T22:25:56.878 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-06-27T22:25:56.879 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.7 query 2024-06-27T22:25:56.879 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-06-27T22:25:57.034 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 3.707005946 + 1.200420323 2024-06-27T22:25:57.034 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=4.907426269 2024-06-27T22:25:57.034 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:100: TEST_snaptrim_stats: test 10 -eq 10 2024-06-27T22:25:57.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: echo '4.907426269 > 0.0' 2024-06-27T22:25:57.035 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: bc 2024-06-27T22:25:57.035 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:25:57.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:103: TEST_snaptrim_stats: teardown td/test-snaptrim-stats 2024-06-27T22:25:57.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-27T22:25:57.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:25:57.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-27T22:25:57.037 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:25:57.037 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:25:57.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:25:57.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:25:57.037 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:25:57.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:25:57.145 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:25:57.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:25:57.147 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:25:57.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:25:57.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:25:57.147 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:25:57.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:25:57.148 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:25:57.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:25:57.149 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:25:57.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:25:57.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:25:57.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-27T22:25:57.182 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:25:57.183 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:25:57.183 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:25:57.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168036 2024-06-27T22:25:57.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:25:57.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:25:57.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-06-27T22:25:57.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-27T22:25:57.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:25:57.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-27T22:25:57.184 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:25:57.184 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:25:57.184 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:25:57.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:25:57.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:25:57.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:25:57.186 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:25:57.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:25:57.187 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:25:57.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:25:57.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:25:57.188 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:25:57.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:25:57.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:25:57.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:25:57.189 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:25:57.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:25:57.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:25:57.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-27T22:25:57.192 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:25:57.192 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:25:57.192 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:25:57.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168036 2024-06-27T22:25:57.193 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:25:57.194 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:25:57.195 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:25:57.195 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:25:57.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:25:57.196 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:25:57.196 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:25:57.197 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:25:57.197 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:25:57.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:25:57.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:25:57.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:25:57.198 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:25:57.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:25:57.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:25:57.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-27T22:25:57.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:25:57.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:25:57.201 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:25:57.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168036 2024-06-27T22:25:57.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:25:57.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:25:57.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-06-27T22:25:57.202 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:25:57.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:25:57.203 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:25:57.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.168036 2024-06-27T22:25:57.205 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:25:57.205 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats_multiple_snaps td/test-snaptrim-stats 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:107: TEST_snaptrim_stats_multiple_snaps: local dir=td/test-snaptrim-stats 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:108: TEST_snaptrim_stats_multiple_snaps: local poolname=test 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:109: TEST_snaptrim_stats_multiple_snaps: local OSDS=3 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:110: TEST_snaptrim_stats_multiple_snaps: local PGNUM=8 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:111: TEST_snaptrim_stats_multiple_snaps: local PGPNUM=8 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:112: TEST_snaptrim_stats_multiple_snaps: local objects=10 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:113: TEST_snaptrim_stats_multiple_snaps: local WAIT_FOR_UPDATE=10 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:115: TEST_snaptrim_stats_multiple_snaps: setup td/test-snaptrim-stats 2024-06-27T22:25:57.206 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-06-27T22:25:57.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-06-27T22:25:57.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-27T22:25:57.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:25:57.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-27T22:25:57.207 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:25:57.207 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:25:57.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:25:57.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:25:57.207 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:25:57.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:25:57.208 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:25:57.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:25:57.208 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:25:57.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:25:57.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:25:57.209 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:25:57.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:25:57.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:25:57.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:25:57.210 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:25:57.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:25:57.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:25:57.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-27T22:25:57.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:25:57.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:25:57.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:25:57.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168036 2024-06-27T22:25:57.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:25:57.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:25:57.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-06-27T22:25:57.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:25:57.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:25:57.214 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:25:57.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.168036 2024-06-27T22:25:57.216 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:25:57.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:25:57.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:25:57.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-06-27T22:25:57.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:116: TEST_snaptrim_stats_multiple_snaps: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-06-27T22:25:57.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-06-27T22:25:57.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:25:57.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:25:57.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:25:57.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-06-27T22:25:57.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-06-27T22:25:57.705 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:25:57.705 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:25:57.705 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:25:57.706 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:25:57.706 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:25:57.706 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:25:57.706 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:25:57.706 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-06-27T22:25:57.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:25:57.910 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:25:57.910 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:25:57.910 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:25:57.910 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:25:57.911 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:25:57.911 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:25:57.911 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:25:57.911 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:25:57.912 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:25:57.912 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:25:57.912 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:25:57.912 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.168036/ceph-mon.a.asok 2024-06-27T22:25:57.912 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:25:57.912 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.168036/ceph-mon.a.asok config get fsid 2024-06-27T22:25:58.029 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:25:58.030 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:25:58.030 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:25:58.030 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:25:58.031 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:25:58.031 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:25:58.031 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:25:58.031 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:25:58.031 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:25:58.031 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:25:58.031 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:25:58.032 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.168036/ceph-mon.a.asok 2024-06-27T22:25:58.032 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:25:58.032 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.168036/ceph-mon.a.asok config get mon_host 2024-06-27T22:25:58.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:117: TEST_snaptrim_stats_multiple_snaps: run_mgr td/test-snaptrim-stats x 2024-06-27T22:25:58.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-06-27T22:25:58.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:25:58.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:25:58.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:25:58.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-06-27T22:25:58.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:25:59.075 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:25:59.075 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:25:59.075 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:25:59.076 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:25:59.076 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:25:59.076 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:25:59.076 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:25:59.076 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:25:59.078 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:25:59.113 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: expr 3 - 1 2024-06-27T22:25:59.113 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: seq 0 2 2024-06-27T22:25:59.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-27T22:25:59.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:25:59.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-27T22:25:59.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:25:59.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:25:59.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:25:59.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-06-27T22:25:59.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-27T22:25:59.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:25:59.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:25:59.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:25:59.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-06-27T22:25:59.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-06-27T22:25:59.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:25:59.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:25:59.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-27T22:25:59.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-27T22:25:59.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:25:59.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:25:59.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:25:59.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:25:59.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:25:59.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:25:59.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-06-27T22:25:59.120 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:25:59.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=519eb23b-ca5b-4ae8-825b-86ef405f2ffc 2024-06-27T22:25:59.121 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 519eb23b-ca5b-4ae8-825b-86ef405f2ffc' 2024-06-27T22:25:59.121 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 519eb23b-ca5b-4ae8-825b-86ef405f2ffc 2024-06-27T22:25:59.122 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:25:59.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD35n1mbv4hCBAAKUPyPzm2fOqHkWVR6B9nMg== 2024-06-27T22:25:59.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD35n1mbv4hCBAAKUPyPzm2fOqHkWVR6B9nMg=="}' 2024-06-27T22:25:59.136 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 519eb23b-ca5b-4ae8-825b-86ef405f2ffc -i td/test-snaptrim-stats/0/new.json 2024-06-27T22:25:59.454 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:25:59.476 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-06-27T22:25:59.477 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQD35n1mbv4hCBAAKUPyPzm2fOqHkWVR6B9nMg== --osd-uuid 519eb23b-ca5b-4ae8-825b-86ef405f2ffc 2024-06-27T22:25:59.512 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:25:59.510+0000 7fbd3d7967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:25:59.530 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:25:59.530+0000 7fbd3d7967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:25:59.531 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:25:59.530+0000 7fbd3d7967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:25:59.532 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:25:59.530+0000 7fbd3d7967c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-27T22:25:59.532 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:25:59.530+0000 7fbd3d7967c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-27T22:25:59.532 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:25:59.530+0000 7fbd3d7967c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-06-27T22:25:59.561 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:25:59.558+0000 7fbd3d7967c0 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-06-27T22:26:06.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-06-27T22:26:06.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:26:06.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:26:06.142 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:26:06.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:26:06.724 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:26:06.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:26:06.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:26:06.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:26:06.725 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:26:06.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:26:06.758 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:06.754+0000 7fbb35a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:26:06.779 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:06.778+0000 7fbb35a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:26:06.780 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:06.778+0000 7fbb35a307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:26:07.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:26:07.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:26:07.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:26:07.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:26:07.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:26:07.209 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:07.209 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:26:07.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:26:07.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:07.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:07.351 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:07.350+0000 7fbb35a307c0 -1 Falling back to public interface 2024-06-27T22:26:07.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:08.713 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:08.714 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:08.714 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:26:08.714 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:26:08.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:08.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:09.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:09.891 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:09.890+0000 7fbb35a307c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:26:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:10.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:10.187 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:26:10.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:26:10.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:10.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:10.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:11.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:11.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:11.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:26:11.779 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:26:11.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:11.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:12.243 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:13.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:13.245 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:13.245 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:26:13.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:26:13.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:13.246 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:13.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:14.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:14.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:14.791 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:26:14.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:26:14.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:14.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:15.250 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:16.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:16.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:16.252 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:26:16.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:26:16.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:16.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:16.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:17.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:17.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:17.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:26:17.757 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:26:17.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:17.758 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:18.208 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:19.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:19.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:19.210 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:26:19.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:26:19.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:19.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:19.766 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:20.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:20.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:20.768 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:26:20.769 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:26:20.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:20.769 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:21.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:22.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:22.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:22.254 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:26:22.255 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:26:22.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:22.255 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:22.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:23.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:23.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:23.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:26:23.806 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:26:23.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:23.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:24.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:25.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:25.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:25.302 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:26:25.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:26:25.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:25.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:26.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:27.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:27.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:27.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:26:27.020 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:26:27.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:27.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:28.012 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:29.014 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:29.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:29.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:26:29.015 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:26:29.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:29.015 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:29.877 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:30.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:30.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:30.879 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:26:30.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:26:30.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:30.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:31.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:32.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:32.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:32.544 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:26:32.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:26:32.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:32.545 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:33.249 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:34.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:34.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:34.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:26:34.252 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:26:34.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:34.252 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:26:34.798 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2502580441,v1:127.0.0.1:6803/2502580441] [v2:127.0.0.1:6804/2502580441,v1:127.0.0.1:6805/2502580441] exists,up 519eb23b-ca5b-4ae8-825b-86ef405f2ffc 2024-06-27T22:26:34.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:26:34.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:26:34.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:26:34.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-27T22:26:34.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:26:34.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-27T22:26:34.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:26:34.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:26:34.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:26:34.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-06-27T22:26:34.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-27T22:26:34.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:26:34.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:26:34.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:26:34.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-06-27T22:26:34.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-06-27T22:26:34.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:26:34.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:26:34.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-27T22:26:34.802 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:26:34.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:26:34.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:26:34.803 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:26:34.803 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:26:34.803 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:26:34.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:26:34.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:26:34.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:26:34.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:26:34.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:26:34.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-27T22:26:34.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-27T22:26:34.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:26:34.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:26:34.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:26:34.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:26:34.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:26:34.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:26:34.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-06-27T22:26:34.805 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:26:34.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8e73881f-71db-429b-a5f0-f292d6a5c864 2024-06-27T22:26:34.805 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 8e73881f-71db-429b-a5f0-f292d6a5c864 2024-06-27T22:26:34.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8e73881f-71db-429b-a5f0-f292d6a5c864' 2024-06-27T22:26:34.806 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:26:34.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAa531msdTsMBAAlzMJKhDK8B9rjAYP7lwsYw== 2024-06-27T22:26:34.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAa531msdTsMBAAlzMJKhDK8B9rjAYP7lwsYw=="}' 2024-06-27T22:26:34.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8e73881f-71db-429b-a5f0-f292d6a5c864 -i td/test-snaptrim-stats/1/new.json 2024-06-27T22:26:35.404 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:26:35.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-06-27T22:26:35.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQAa531msdTsMBAAlzMJKhDK8B9rjAYP7lwsYw== --osd-uuid 8e73881f-71db-429b-a5f0-f292d6a5c864 2024-06-27T22:26:35.456 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:35.454+0000 7f54079537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:26:35.467 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:35.466+0000 7f54079537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:26:35.468 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:35.466+0000 7f54079537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:26:35.469 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:35.466+0000 7f54079537c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-27T22:26:35.469 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:35.466+0000 7f54079537c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-27T22:26:35.469 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:35.466+0000 7f54079537c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-06-27T22:26:35.497 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:35.494+0000 7f54079537c0 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-06-27T22:26:42.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-06-27T22:26:42.145 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:26:42.145 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:26:42.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:26:42.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:26:42.731 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:26:42.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:26:42.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:26:42.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:26:42.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:26:42.732 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:26:42.764 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:42.762+0000 7f6621a437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:26:42.794 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:42.790+0000 7f6621a437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:26:42.794 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:42.790+0000 7f6621a437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:26:43.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:26:43.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:26:43.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:26:43.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:26:43.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:26:43.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:43.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:26:43.187 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:26:43.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:43.188 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:26:43.388 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:43.386+0000 7f6621a437c0 -1 Falling back to public interface 2024-06-27T22:26:43.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:44.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:44.632 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:44.632 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:26:44.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:26:44.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:44.633 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:26:45.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:46.025 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:26:46.022+0000 7f6621a437c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:26:46.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:46.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:46.117 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:26:46.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:26:46.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:46.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:26:46.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:47.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:47.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:47.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:26:47.806 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:26:47.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:47.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:26:48.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:49.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:49.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:49.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:26:49.370 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:26:49.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:49.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:26:49.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:50.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:50.951 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:50.951 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:26:50.951 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:26:50.951 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:50.952 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:26:51.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:52.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:52.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:52.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:26:52.388 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:26:52.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:52.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:26:52.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:53.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:53.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:53.932 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:26:53.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:26:53.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:53.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:26:54.396 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:55.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:55.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:55.398 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:26:55.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:26:55.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:55.399 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:26:55.898 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:56.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:56.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:56.900 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:26:56.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:26:56.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:56.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:26:57.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:58.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:58.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:58.404 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:26:58.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:26:58.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:58.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:26:58.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:26:59.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:26:59.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:26:59.962 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:26:59.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:26:59.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:26:59.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:27:00.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:01.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:01.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:01.710 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:27:01.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:27:01.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:01.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:27:02.380 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:03.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:03.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:03.382 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:27:03.383 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:27:03.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:03.383 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:27:04.754 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:05.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:05.755 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:05.756 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:27:05.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:27:05.756 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:05.757 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:27:06.714 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:07.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:07.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:07.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:27:07.717 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:27:07.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:07.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:27:08.939 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:09.940 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:09.941 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:09.941 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:27:09.941 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:27:09.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:09.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:27:10.965 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:11.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:11.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:11.968 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:27:11.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:27:11.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:11.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:27:12.510 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2932675739,v1:127.0.0.1:6811/2932675739] [v2:127.0.0.1:6812/2932675739,v1:127.0.0.1:6813/2932675739] exists,up 8e73881f-71db-429b-a5f0-f292d6a5c864 2024-06-27T22:27:12.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:27:12.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:27:12.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:27:12.515 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-06-27T22:27:12.516 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:27:12.517 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:27:12.518 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d099b4d3-9d07-4245-bcfd-ac5fd99aad5f 2024-06-27T22:27:12.519 INFO:tasks.workunit.client.0.smithi042.stdout:add osd2 d099b4d3-9d07-4245-bcfd-ac5fd99aad5f 2024-06-27T22:27:12.519 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d099b4d3-9d07-4245-bcfd-ac5fd99aad5f' 2024-06-27T22:27:12.519 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:27:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBA531mljINIBAAR7XlmP54w3hPY0klQyCRQQ== 2024-06-27T22:27:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBA531mljINIBAAR7XlmP54w3hPY0klQyCRQQ=="}' 2024-06-27T22:27:12.537 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d099b4d3-9d07-4245-bcfd-ac5fd99aad5f -i td/test-snaptrim-stats/2/new.json 2024-06-27T22:27:13.041 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:27:13.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-06-27T22:27:13.058 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBA531mljINIBAAR7XlmP54w3hPY0klQyCRQQ== --osd-uuid d099b4d3-9d07-4245-bcfd-ac5fd99aad5f 2024-06-27T22:27:13.085 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:13.082+0000 7f6deca587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:27:13.112 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:13.110+0000 7f6deca587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:27:13.113 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:13.110+0000 7f6deca587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:27:13.114 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:13.110+0000 7f6deca587c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-27T22:27:13.114 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:13.110+0000 7f6deca587c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-27T22:27:13.114 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:13.110+0000 7f6deca587c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-06-27T22:27:13.151 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:13.150+0000 7f6deca587c0 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-06-27T22:27:19.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-06-27T22:27:19.727 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:27:19.729 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd2 key to auth repository 2024-06-27T22:27:19.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-27T22:27:19.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:27:20.347 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T22:27:20.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-27T22:27:20.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6ff7a045-a6ff-4928-8bc8-fee42a53f664 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.168036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-06-27T22:27:20.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:27:20.348 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:27:20.349 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:27:20.379 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:20.378+0000 7f39695417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:27:20.405 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:20.402+0000 7f39695417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:27:20.407 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:20.406+0000 7f39695417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:27:20.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-27T22:27:20.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:27:20.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T22:27:20.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:27:20.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:27:20.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:20.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:27:20.792 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:27:20.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:20.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:20.995 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:20.994+0000 7f39695417c0 -1 Falling back to public interface 2024-06-27T22:27:21.270 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:22.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:22.272 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:22.273 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:27:22.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:27:22.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:22.273 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:22.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:23.613 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:27:23.610+0000 7f39695417c0 -1 osd.2 0 log_to_monitors true 2024-06-27T22:27:23.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:23.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:23.717 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:27:23.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:27:23.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:23.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:24.481 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:25.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:25.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:25.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:27:25.483 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:27:25.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:25.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:25.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:26.930 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:26.930 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:26.930 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:27:26.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:27:26.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:26.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:27.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:28.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:28.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:28.375 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:27:28.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:27:28.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:28.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:28.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:29.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:29.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:29.900 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:27:29.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:27:29.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:29.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:30.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:31.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:31.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:31.405 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:27:31.406 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:27:31.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:31.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:31.903 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:32.904 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:32.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:32.905 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:27:32.905 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:27:32.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:32.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:33.368 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:34.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:34.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:34.370 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:27:34.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:27:34.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:34.370 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:34.897 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:35.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:35.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:35.899 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:27:35.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:27:35.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:35.900 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:36.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:37.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:37.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:37.409 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:27:37.410 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:27:37.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:37.410 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:38.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:39.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:39.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:39.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:27:39.180 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:27:39.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:39.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:40.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:41.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:41.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:41.171 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:27:41.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:27:41.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:41.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:41.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:42.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:42.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:42.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:27:42.961 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:27:42.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:42.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:43.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:44.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:44.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:44.807 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:27:44.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:27:44.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:44.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:45.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:46.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:46.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:46.672 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:27:46.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:27:46.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:46.673 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:47.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:48.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:48.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:48.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:27:48.303 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:27:48.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:48.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:48.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:27:49.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:27:49.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:27:49.809 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:27:49.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:27:49.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:27:49.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:27:50.280 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2629644534,v1:127.0.0.1:6819/2629644534] [v2:127.0.0.1:6820/2629644534,v1:127.0.0.1:6821/2629644534] exists,up d099b4d3-9d07-4245-bcfd-ac5fd99aad5f 2024-06-27T22:27:50.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:27:50.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:27:50.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:27:50.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:124: TEST_snaptrim_stats_multiple_snaps: ceph osd set noscrub 2024-06-27T22:27:51.066 INFO:tasks.workunit.client.0.smithi042.stderr:noscrub is set 2024-06-27T22:27:51.086 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:125: TEST_snaptrim_stats_multiple_snaps: ceph osd set nodeep-scrub 2024-06-27T22:27:52.013 INFO:tasks.workunit.client.0.smithi042.stderr:nodeep-scrub is set 2024-06-27T22:27:52.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:128: TEST_snaptrim_stats_multiple_snaps: create_pool test 8 8 2024-06-27T22:27:52.036 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-06-27T22:27:52.623 INFO:tasks.workunit.client.0.smithi042.stderr:pool 'test' already exists 2024-06-27T22:27:52.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-27T22:27:53.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:129: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-06-27T22:27:53.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:27:53.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:27:53.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:27:53.641 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:27:53.643 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:27:53.643 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:27:53.643 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:27:53.643 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:27:53.643 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:27:53.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:27:53.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:27:53.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:27:53.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:27:53.771 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:27:53.771 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:27:54.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T22:27:54.314 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T22:27:54.314 INFO:tasks.workunit.client.0.smithi042.stderr:2' 2024-06-27T22:27:54.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:27:54.314 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:27:54.314 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:27:54.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836502 2024-06-27T22:27:54.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836502 2024-06-27T22:27:54.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502' 2024-06-27T22:27:54.476 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:27:54.476 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T22:27:54.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672982 2024-06-27T22:27:54.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672982 2024-06-27T22:27:54.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672982' 2024-06-27T22:27:54.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:27:54.712 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T22:27:54.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509445 2024-06-27T22:27:54.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509445 2024-06-27T22:27:54.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672982 2-64424509445' 2024-06-27T22:27:54.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:27:54.877 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836502 2024-06-27T22:27:54.877 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:27:54.878 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:27:54.878 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-06-27T22:27:54.878 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:27:54.879 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836502 2024-06-27T22:27:54.880 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836502 2024-06-27T22:27:54.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-06-27T22:27:54.880 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:27:55.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836500 -lt 21474836502 2024-06-27T22:27:55.302 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:27:56.303 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:27:56.304 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:27:56.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836502 2024-06-27T22:27:56.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:27:56.743 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672982 2024-06-27T22:27:56.743 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:27:56.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T22:27:56.745 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-06-27T22:27:56.745 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:27:56.746 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672982 2024-06-27T22:27:56.746 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-06-27T22:27:56.747 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672982 2024-06-27T22:27:56.747 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T22:27:57.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672982 2024-06-27T22:27:57.251 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:27:57.252 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509445 2024-06-27T22:27:57.252 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:27:57.253 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T22:27:57.254 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509445 2024-06-27T22:27:57.254 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:27:57.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509445 2024-06-27T22:27:57.256 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509445 2024-06-27T22:27:57.256 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-06-27T22:27:57.256 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T22:27:57.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509445 -lt 64424509445 2024-06-27T22:27:57.745 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:27:57.745 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:27:57.745 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:27:58.356 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-06-27T22:27:58.356 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:27:58.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:27:58.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:27:58.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:27:58.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:27:58.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:27:58.357 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:27:58.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-27T22:27:58.812 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:27:58.812 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:27:58.812 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:27:59.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-06-27T22:27:59.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:27:59.408 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:27:59.408 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: ceph osd dump 2024-06-27T22:27:59.409 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: awk '{ print $2 }' 2024-06-27T22:27:59.409 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: grep '^pool.*['\'']test['\'']' 2024-06-27T22:27:59.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: poolid=1 2024-06-27T22:27:59.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:133: TEST_snaptrim_stats_multiple_snaps: local TESTDATA=testdata.0 2024-06-27T22:27:59.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:134: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.0 bs=4096 count=1 2024-06-27T22:27:59.867 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records in 2024-06-27T22:27:59.867 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records out 2024-06-27T22:27:59.867 INFO:tasks.workunit.client.0.smithi042.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000130131 s, 31.5 MB/s 2024-06-27T22:27:59.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-06-27T22:27:59.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:27:59.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.0 2024-06-27T22:28:00.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:00.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.0 2024-06-27T22:28:00.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:00.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.0 2024-06-27T22:28:00.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:00.498 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.0 2024-06-27T22:28:00.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:00.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.0 2024-06-27T22:28:00.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:00.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.0 2024-06-27T22:28:00.993 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:00.994 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.0 2024-06-27T22:28:01.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:01.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.0 2024-06-27T22:28:01.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:01.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.0 2024-06-27T22:28:01.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:01.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.0 2024-06-27T22:28:01.671 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:139: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.0 2024-06-27T22:28:01.672 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:142: TEST_snaptrim_stats_multiple_snaps: NUMSNAPS=2 2024-06-27T22:28:01.673 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-06-27T22:28:01.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-06-27T22:28:01.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap1 2024-06-27T22:28:01.864 INFO:tasks.workunit.client.0.smithi042.stdout:created pool test snap snap1 2024-06-27T22:28:01.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.1 2024-06-27T22:28:01.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-06-27T22:28:01.868 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records in 2024-06-27T22:28:01.868 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records out 2024-06-27T22:28:01.868 INFO:tasks.workunit.client.0.smithi042.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000184378 s, 22.2 MB/s 2024-06-27T22:28:01.869 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-06-27T22:28:01.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:01.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.1 2024-06-27T22:28:02.199 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:02.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.1 2024-06-27T22:28:02.384 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:02.384 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.1 2024-06-27T22:28:02.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:02.544 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.1 2024-06-27T22:28:02.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:02.704 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.1 2024-06-27T22:28:02.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:02.880 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.1 2024-06-27T22:28:03.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:03.056 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.1 2024-06-27T22:28:03.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:03.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.1 2024-06-27T22:28:03.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:03.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.1 2024-06-27T22:28:03.542 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:03.543 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.1 2024-06-27T22:28:03.710 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.1 2024-06-27T22:28:03.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-06-27T22:28:03.712 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap2 2024-06-27T22:28:03.877 INFO:tasks.workunit.client.0.smithi042.stdout:created pool test snap snap2 2024-06-27T22:28:03.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.2 2024-06-27T22:28:03.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-06-27T22:28:03.882 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records in 2024-06-27T22:28:03.882 INFO:tasks.workunit.client.0.smithi042.stderr:1+0 records out 2024-06-27T22:28:03.882 INFO:tasks.workunit.client.0.smithi042.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000247625 s, 16.5 MB/s 2024-06-27T22:28:03.883 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-06-27T22:28:03.884 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:03.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.2 2024-06-27T22:28:04.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:04.301 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.2 2024-06-27T22:28:04.482 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:04.482 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.2 2024-06-27T22:28:04.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:04.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.2 2024-06-27T22:28:04.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:04.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.2 2024-06-27T22:28:04.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:04.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.2 2024-06-27T22:28:05.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:05.119 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.2 2024-06-27T22:28:05.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:05.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.2 2024-06-27T22:28:05.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:05.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.2 2024-06-27T22:28:05.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-06-27T22:28:05.639 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.2 2024-06-27T22:28:05.798 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.2 2024-06-27T22:28:05.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:156: TEST_snaptrim_stats_multiple_snaps: local total_objects_trimmed=0 2024-06-27T22:28:05.800 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-06-27T22:28:05.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-06-27T22:28:05.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap1 2024-06-27T22:28:06.057 INFO:tasks.workunit.client.0.smithi042.stdout:removed pool test snap snap1 2024-06-27T22:28:06.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-06-27T22:28:06.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:28:06.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:28:06.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:28:06.062 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:28:06.063 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:28:06.063 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:28:06.064 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:28:06.064 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:28:06.064 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:28:06.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:28:06.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:28:06.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:28:06.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:28:06.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:28:06.190 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:28:06.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T22:28:06.676 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T22:28:06.676 INFO:tasks.workunit.client.0.smithi042.stderr:2' 2024-06-27T22:28:06.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:28:06.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:28:06.677 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:28:06.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836507 2024-06-27T22:28:06.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836507 2024-06-27T22:28:06.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507' 2024-06-27T22:28:06.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:28:06.956 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T22:28:07.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672986 2024-06-27T22:28:07.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672986 2024-06-27T22:28:07.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672986' 2024-06-27T22:28:07.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:28:07.159 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T22:28:07.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509449 2024-06-27T22:28:07.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509449 2024-06-27T22:28:07.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672986 2-64424509449' 2024-06-27T22:28:07.334 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:28:07.334 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836507 2024-06-27T22:28:07.335 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:28:07.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:28:07.336 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836507 2024-06-27T22:28:07.336 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:28:07.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836507 2024-06-27T22:28:07.338 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836507 2024-06-27T22:28:07.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-06-27T22:28:07.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:28:07.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836507 2024-06-27T22:28:07.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-06-27T22:28:08.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-27T22:28:08.764 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:28:09.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836507 2024-06-27T22:28:09.295 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:28:09.296 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672986 2024-06-27T22:28:09.296 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:28:09.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T22:28:09.298 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672986 2024-06-27T22:28:09.298 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:28:09.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672986 2024-06-27T22:28:09.300 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672986 2024-06-27T22:28:09.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-06-27T22:28:09.300 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T22:28:09.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672987 -lt 42949672986 2024-06-27T22:28:09.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:28:09.738 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509449 2024-06-27T22:28:09.738 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:28:09.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T22:28:09.740 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509449 2024-06-27T22:28:09.740 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:28:09.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509449 2024-06-27T22:28:09.741 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509449 2024-06-27T22:28:09.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-06-27T22:28:09.742 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T22:28:10.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509449 2024-06-27T22:28:10.163 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:28:10.163 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:28:10.164 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:28:10.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-06-27T22:28:10.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:28:10.752 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:28:10.752 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:28:10.752 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:28:10.752 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:28:10.752 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:28:10.752 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:28:11.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-27T22:28:11.183 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:28:11.184 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:28:11.184 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:28:11.777 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-06-27T22:28:11.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:28:11.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:28:11.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-06-27T22:28:21.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-06-27T22:28:21.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-06-27T22:28:21.780 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-06-27T22:28:21.782 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-06-27T22:28:21.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:21.784 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-06-27T22:28:21.785 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:21.785 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-06-27T22:28:21.958 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-06-27T22:28:21.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-06-27T22:28:21.960 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:21.960 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-06-27T22:28:21.960 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:22.119 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 1.141911489 2024-06-27T22:28:22.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.141911489 2024-06-27T22:28:22.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:22.120 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-06-27T22:28:22.121 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:22.121 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-06-27T22:28:22.297 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-06-27T22:28:22.298 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-06-27T22:28:22.299 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:22.299 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-06-27T22:28:22.299 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:22.454 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.141911489 + 0.461964919 2024-06-27T22:28:22.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.603876408 2024-06-27T22:28:22.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:22.454 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-06-27T22:28:22.455 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-06-27T22:28:22.455 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:22.612 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-06-27T22:28:22.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-06-27T22:28:22.613 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:22.614 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-06-27T22:28:22.614 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:22.771 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.603876408 + 0.110660306 2024-06-27T22:28:22.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.714536714 2024-06-27T22:28:22.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:22.772 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-06-27T22:28:22.773 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-06-27T22:28:22.773 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:22.931 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-06-27T22:28:22.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-06-27T22:28:22.933 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:22.933 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-06-27T22:28:22.933 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:23.083 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.714536714 + 0.473112923 2024-06-27T22:28:23.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.187649637 2024-06-27T22:28:23.083 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:23.084 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-06-27T22:28:23.084 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-06-27T22:28:23.084 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:23.236 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-06-27T22:28:23.237 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-06-27T22:28:23.237 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:23.238 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-06-27T22:28:23.238 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:23.396 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.187649637 + 0.427051399 2024-06-27T22:28:23.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.614701036 2024-06-27T22:28:23.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:23.397 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-06-27T22:28:23.398 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-06-27T22:28:23.398 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:23.548 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-06-27T22:28:23.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-06-27T22:28:23.549 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:23.550 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-06-27T22:28:23.550 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:23.708 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.614701036 + 0.201507723 2024-06-27T22:28:23.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.816208759 2024-06-27T22:28:23.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:23.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-06-27T22:28:23.710 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:23.710 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-06-27T22:28:23.870 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-06-27T22:28:23.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-06-27T22:28:23.871 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:23.872 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-06-27T22:28:23.872 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:24.026 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.816208759 + 0.974263345 2024-06-27T22:28:24.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.790472104 2024-06-27T22:28:24.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:24.027 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-06-27T22:28:24.028 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-06-27T22:28:24.028 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:24.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-06-27T22:28:24.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-06-27T22:28:24.213 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:24.214 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-06-27T22:28:24.214 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:24.375 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.790472104 + 0.973977039 2024-06-27T22:28:24.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.764449143 2024-06-27T22:28:24.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-06-27T22:28:24.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '4.764449143 > 0.0' 2024-06-27T22:28:24.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:24.377 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:28:24.377 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 0 + 10 2024-06-27T22:28:24.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=10 2024-06-27T22:28:24.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-06-27T22:28:24.378 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap2 2024-06-27T22:28:24.561 INFO:tasks.workunit.client.0.smithi042.stdout:removed pool test snap snap2 2024-06-27T22:28:24.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-06-27T22:28:24.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-27T22:28:24.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-27T22:28:24.565 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-27T22:28:24.565 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-27T22:28:24.566 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-27T22:28:24.566 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-27T22:28:24.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-27T22:28:24.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-27T22:28:24.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-27T22:28:24.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-27T22:28:24.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-27T22:28:24.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-27T22:28:24.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-27T22:28:24.685 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-06-27T22:28:24.685 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-06-27T22:28:25.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-06-27T22:28:25.247 INFO:tasks.workunit.client.0.smithi042.stderr:1 2024-06-27T22:28:25.247 INFO:tasks.workunit.client.0.smithi042.stderr:2' 2024-06-27T22:28:25.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-06-27T22:28:25.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:28:25.248 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-27T22:28:25.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836513 2024-06-27T22:28:25.491 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836513 2024-06-27T22:28:25.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513' 2024-06-27T22:28:25.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:28:25.492 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-27T22:28:25.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672992 2024-06-27T22:28:25.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672992 2024-06-27T22:28:25.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-42949672992' 2024-06-27T22:28:25.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-06-27T22:28:25.763 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-27T22:28:25.927 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-06-27T22:28:25.927 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-06-27T22:28:25.927 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509455' 2024-06-27T22:28:25.927 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:28:25.928 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836513 2024-06-27T22:28:25.928 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:28:25.929 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-06-27T22:28:25.930 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836513 2024-06-27T22:28:25.930 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:28:25.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836513 2024-06-27T22:28:25.931 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.0 seq 21474836513 2024-06-27T22:28:25.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-06-27T22:28:25.932 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-27T22:28:26.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836513 -lt 21474836513 2024-06-27T22:28:26.371 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:28:26.372 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672992 2024-06-27T22:28:26.372 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:28:26.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-06-27T22:28:26.374 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672992 2024-06-27T22:28:26.375 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:28:26.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672992 2024-06-27T22:28:26.376 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.1 seq 42949672992 2024-06-27T22:28:26.376 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-06-27T22:28:26.376 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-27T22:28:26.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672992 -lt 42949672992 2024-06-27T22:28:26.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-06-27T22:28:26.862 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-06-27T22:28:26.862 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-06-27T22:28:26.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-06-27T22:28:26.865 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-06-27T22:28:26.865 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-06-27T22:28:26.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-06-27T22:28:26.866 INFO:tasks.workunit.client.0.smithi042.stdout:waiting osd.2 seq 64424509455 2024-06-27T22:28:26.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-06-27T22:28:26.867 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-27T22:28:27.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509455 2024-06-27T22:28:27.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-27T22:28:27.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:28:27.338 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:28:27.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-06-27T22:28:27.915 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-27T22:28:27.915 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-27T22:28:27.915 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-27T22:28:27.915 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-27T22:28:27.915 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-27T22:28:27.915 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-27T22:28:27.916 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-27T22:28:28.347 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-06-27T22:28:28.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-27T22:28:28.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-27T22:28:28.348 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-27T22:28:28.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-06-27T22:28:28.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-27T22:28:28.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-27T22:28:28.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-06-27T22:28:38.910 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-06-27T22:28:38.910 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-06-27T22:28:38.911 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-06-27T22:28:38.913 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-06-27T22:28:38.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:38.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-06-27T22:28:38.916 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-06-27T22:28:38.916 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:39.086 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-06-27T22:28:39.087 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-06-27T22:28:39.087 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:39.088 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-06-27T22:28:39.089 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:39.247 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.394288665 2024-06-27T22:28:39.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.394288665 2024-06-27T22:28:39.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:39.247 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-06-27T22:28:39.248 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-06-27T22:28:39.248 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:39.401 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-06-27T22:28:39.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-06-27T22:28:39.402 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:39.403 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-06-27T22:28:39.403 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:39.558 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .394288665 + 0.420332008 2024-06-27T22:28:39.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.814620673 2024-06-27T22:28:39.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:39.559 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-06-27T22:28:39.560 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-06-27T22:28:39.560 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:39.723 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-06-27T22:28:39.723 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-06-27T22:28:39.724 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:39.725 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-06-27T22:28:39.725 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:39.884 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .814620673 + 0.117556527 2024-06-27T22:28:39.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.932177200 2024-06-27T22:28:39.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:39.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-06-27T22:28:39.886 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-06-27T22:28:39.886 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:40.044 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-06-27T22:28:40.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-06-27T22:28:40.044 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:40.045 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-06-27T22:28:40.045 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:40.212 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .932177200 + 0.814305835 2024-06-27T22:28:40.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.746483035 2024-06-27T22:28:40.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:40.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-06-27T22:28:40.214 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-06-27T22:28:40.214 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:40.373 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-06-27T22:28:40.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-06-27T22:28:40.375 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:40.375 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-06-27T22:28:40.376 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:40.533 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.746483035 + 0.260744532 2024-06-27T22:28:40.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.007227567 2024-06-27T22:28:40.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:40.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-06-27T22:28:40.534 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-06-27T22:28:40.534 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:40.700 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-06-27T22:28:40.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-06-27T22:28:40.701 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:40.701 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-06-27T22:28:40.701 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:40.852 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.007227567 + 0.4200429 2024-06-27T22:28:40.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.427270467 2024-06-27T22:28:40.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:40.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-06-27T22:28:40.854 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-06-27T22:28:40.854 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:41.008 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-06-27T22:28:41.008 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-06-27T22:28:41.009 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:41.010 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-06-27T22:28:41.010 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:41.167 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.427270467 + 0.520364108 2024-06-27T22:28:41.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.947634575 2024-06-27T22:28:41.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-06-27T22:28:41.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-06-27T22:28:41.169 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-06-27T22:28:41.169 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-06-27T22:28:41.326 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-06-27T22:28:41.327 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-06-27T22:28:41.327 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:41.328 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-06-27T22:28:41.328 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-06-27T22:28:41.483 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.947634575 + 0.940478916 2024-06-27T22:28:41.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.888113491 2024-06-27T22:28:41.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-06-27T22:28:41.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '3.888113491 > 0.0' 2024-06-27T22:28:41.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-06-27T22:28:41.485 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:28:41.485 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 10 + 10 2024-06-27T22:28:41.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=20 2024-06-27T22:28:41.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:179: TEST_snaptrim_stats_multiple_snaps: test 20 -eq 20 2024-06-27T22:28:41.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:181: TEST_snaptrim_stats_multiple_snaps: teardown td/test-snaptrim-stats 2024-06-27T22:28:41.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-27T22:28:41.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:28:41.487 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-27T22:28:41.487 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:28:41.488 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:28:41.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:28:41.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:28:41.488 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:28:41.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:28:41.797 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:28:41.798 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:28:41.799 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:28:41.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:28:41.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:28:41.800 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:28:41.801 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:28:41.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:28:41.802 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:28:41.803 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:28:41.804 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:28:41.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:28:41.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-27T22:28:41.848 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:28:41.848 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:41.848 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:28:41.848 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168036 2024-06-27T22:28:41.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:28:41.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:28:41.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-06-27T22:28:41.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-27T22:28:41.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:28:41.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-27T22:28:41.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:28:41.850 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:28:41.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:28:41.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:28:41.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:28:41.853 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:28:41.853 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:28:41.854 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:28:41.854 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:28:41.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:28:41.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:28:41.856 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:28:41.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:28:41.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:28:41.857 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:28:41.858 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:28:41.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:28:41.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:28:41.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-27T22:28:41.861 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:28:41.861 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:41.861 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:28:41.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168036 2024-06-27T22:28:41.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:28:41.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:28:41.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-06-27T22:28:41.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/test-snaptrim-stats 0 2024-06-27T22:28:41.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-06-27T22:28:41.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-27T22:28:41.863 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-06-27T22:28:41.864 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:28:41.864 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:28:41.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:28:41.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:28:41.864 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:28:41.866 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:28:41.866 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:28:41.867 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:28:41.868 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:28:41.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:28:41.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:28:41.869 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:28:41.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:28:41.870 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:28:41.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:28:41.871 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:28:41.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:28:41.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-27T22:28:41.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-06-27T22:28:41.875 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:28:41.875 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:41.875 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168036 2024-06-27T22:28:41.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168036 2024-06-27T22:28:41.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:28:41.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:28:41.876 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-06-27T22:28:41.877 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-27T22:28:41.878 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-27T22:28:41.890 INFO:tasks.workunit:Running workunit misc/ver-health.sh... 2024-06-27T22:28:41.891 DEBUG:teuthology.orchestra.run.smithi042:workunit test misc/ver-health.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh 2024-06-27T22:28:41.950 INFO:tasks.workunit.client.0.smithi042.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-27T22:28:41.956 INFO:tasks.workunit.client.0.smithi042.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-27T22:28:41.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T22:28:41.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-27T22:28:41.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-06-27T22:28:41.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-06-27T22:28:41.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-06-27T22:28:41.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-06-27T22:28:41.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-06-27T22:28:41.956 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-06-27T22:28:41.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/ver-health 2024-06-27T22:28:41.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:20: run: local dir=td/ver-health 2024-06-27T22:28:41.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:21: run: shift 2024-06-27T22:28:41.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: export CEPH_MON_A=127.0.0.1:7165 2024-06-27T22:28:41.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: CEPH_MON_A=127.0.0.1:7165 2024-06-27T22:28:41.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: export CEPH_MON_B=127.0.0.1:7166 2024-06-27T22:28:41.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: CEPH_MON_B=127.0.0.1:7166 2024-06-27T22:28:41.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:25: run: export CEPH_ARGS 2024-06-27T22:28:41.957 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: uuidgen 2024-06-27T22:28:41.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: CEPH_ARGS+='--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none ' 2024-06-27T22:28:41.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:27: run: CEPH_ARGS+='--mon-host= ' 2024-06-27T22:28:41.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:28: run: CEPH_ARGS+='--mon_health_to_clog_tick_interval=1.0 ' 2024-06-27T22:28:41.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: export 'ORIG_CEPH_ARGS=--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-06-27T22:28:41.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: ORIG_CEPH_ARGS='--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-06-27T22:28:41.958 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: set 2024-06-27T22:28:41.958 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-27T22:28:41.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: local 'funcs=TEST_check_version_health_1 2024-06-27T22:28:41.960 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_check_version_health_2 2024-06-27T22:28:41.960 INFO:tasks.workunit.client.0.smithi042.stderr:TEST_check_version_health_3' 2024-06-27T22:28:41.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-06-27T22:28:41.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-06-27T22:28:41.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-06-27T22:28:41.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-06-27T22:28:41.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-27T22:28:41.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:28:41.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-27T22:28:41.961 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:28:41.961 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:28:41.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:28:41.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:28:41.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:28:41.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:28:41.962 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:28:41.963 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:28:41.963 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:28:41.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:28:41.964 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:28:41.965 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:28:41.965 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:28:41.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:28:41.966 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:28:41.966 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:28:41.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:28:41.968 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:28:41.968 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-27T22:28:41.969 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:28:41.970 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:41.970 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:28:41.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.183441 2024-06-27T22:28:41.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:28:41.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:28:41.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-06-27T22:28:41.973 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:28:41.973 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:41.973 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:28:41.973 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.183441 2024-06-27T22:28:41.976 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:28:41.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-27T22:28:41.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-27T22:28:41.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:28:41.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-06-27T22:28:41.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_1 td/ver-health 2024-06-27T22:28:41.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:68: TEST_check_version_health_1: local dir=td/ver-health 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:71: TEST_check_version_health_1: CEPH_ARGS='--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:73: TEST_check_version_health_1: setup td/ver-health 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:28:41.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:28:41.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:28:41.978 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:28:41.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:28:41.979 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:28:41.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:28:41.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:28:41.981 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:28:41.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:28:41.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:28:41.982 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:28:41.982 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:28:41.983 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:28:41.984 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:28:41.984 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-27T22:28:41.985 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:28:41.985 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:41.986 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:28:41.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.183441 2024-06-27T22:28:41.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:28:41.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:28:41.987 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-06-27T22:28:41.989 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:28:41.989 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:41.989 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:28:41.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.183441 2024-06-27T22:28:41.990 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:28:41.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:28:41.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:28:41.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-06-27T22:28:41.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:76: TEST_check_version_health_1: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-27T22:28:41.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-06-27T22:28:41.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:28:41.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:28:41.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:28:41.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-06-27T22:28:41.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-27T22:28:42.476 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:28:42.476 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:28:42.476 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:28:42.477 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:28:42.477 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:42.477 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:28:42.477 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:28:42.477 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-27T22:28:42.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:28:42.698 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:28:42.698 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:28:42.699 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:28:42.699 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:28:42.699 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:28:42.699 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:28:42.699 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:28:42.699 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:28:42.700 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:28:42.700 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:42.700 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:28:42.700 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.183441/ceph-mon.a.asok 2024-06-27T22:28:42.700 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:28:42.700 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.183441/ceph-mon.a.asok config get fsid 2024-06-27T22:28:42.820 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:28:42.820 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:28:42.820 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:28:42.820 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:28:42.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:28:42.821 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:28:42.821 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:28:42.821 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:28:42.822 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:28:42.822 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:42.822 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:28:42.823 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.183441/ceph-mon.a.asok 2024-06-27T22:28:42.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:28:42.823 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.183441/ceph-mon.a.asok config get mon_host 2024-06-27T22:28:42.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:77: TEST_check_version_health_1: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-27T22:28:42.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-06-27T22:28:42.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:28:42.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-27T22:28:42.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:28:42.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-06-27T22:28:42.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-27T22:28:43.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:28:43.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:28:43.844 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:28:43.845 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:28:43.845 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:43.845 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:28:43.845 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:28:43.845 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-27T22:28:44.047 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:28:44.048 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-27T22:28:44.048 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:28:44.048 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T22:28:44.048 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:28:44.049 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:28:44.049 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T22:28:44.049 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T22:28:44.049 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T22:28:44.050 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:28:44.050 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:44.050 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:28:44.050 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.183441/ceph-mon.b.asok 2024-06-27T22:28:44.050 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:28:44.051 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.183441/ceph-mon.b.asok config get fsid 2024-06-27T22:28:44.182 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-27T22:28:44.182 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:28:44.182 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T22:28:44.182 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:28:44.183 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:28:44.183 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T22:28:44.183 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T22:28:44.183 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T22:28:44.183 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:28:44.183 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:44.183 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:28:44.184 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.183441/ceph-mon.b.asok 2024-06-27T22:28:44.184 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:28:44.184 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.183441/ceph-mon.b.asok config get mon_host 2024-06-27T22:28:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:78: TEST_check_version_health_1: run_osd td/ver-health 0 2024-06-27T22:28:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-27T22:28:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:28:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:28:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:28:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-06-27T22:28:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-27T22:28:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:28:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:28:44.304 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:28:44.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-06-27T22:28:44.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-06-27T22:28:44.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:28:44.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:28:44.305 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-27T22:28:44.305 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:28:44.305 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:28:44.305 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:28:44.305 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:28:44.305 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:28:44.305 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:28:44.306 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:28:44.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:28:44.306 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-06-27T22:28:44.307 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:28:44.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=011120a2-2b59-46a3-bdf1-39052865b83b 2024-06-27T22:28:44.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 011120a2-2b59-46a3-bdf1-39052865b83b' 2024-06-27T22:28:44.308 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 011120a2-2b59-46a3-bdf1-39052865b83b 2024-06-27T22:28:44.309 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:28:44.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCc531mvUx1ExAAQD9F4/zudqkc536bkE9SmQ== 2024-06-27T22:28:44.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCc531mvUx1ExAAQD9F4/zudqkc536bkE9SmQ=="}' 2024-06-27T22:28:44.326 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 011120a2-2b59-46a3-bdf1-39052865b83b -i td/ver-health/0/new.json 2024-06-27T22:28:50.718 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:28:50.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-06-27T22:28:50.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCc531mvUx1ExAAQD9F4/zudqkc536bkE9SmQ== --osd-uuid 011120a2-2b59-46a3-bdf1-39052865b83b 2024-06-27T22:28:50.778 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:28:50.777+0000 7f85ba8487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:28:50.781 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:28:50.777+0000 7f85ba8487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:28:50.783 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:28:50.781+0000 7f85ba8487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:28:50.783 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:28:50.781+0000 7f85ba8487c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-27T22:28:50.783 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:28:50.781+0000 7f85ba8487c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-27T22:28:50.783 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:28:50.781+0000 7f85ba8487c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-27T22:28:50.879 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:28:50.877+0000 7f85ba8487c0 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-06-27T22:28:56.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-06-27T22:28:56.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:28:56.977 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:28:56.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:28:56.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:28:57.235 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:28:57.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:28:57.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:28:57.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:28:57.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:28:57.236 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:28:57.270 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:28:57.265+0000 7f97b330d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:28:57.272 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:28:57.269+0000 7f97b330d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:28:57.274 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:28:57.269+0000 7f97b330d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:28:57.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:28:57.587 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:28:57.587 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:28:57.587 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:28:57.587 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:28:57.587 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:28:57.587 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:28:57.587 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:28:57.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:28:57.588 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:28:57.828 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:28:57.825+0000 7f97b330d7c0 -1 Falling back to public interface 2024-06-27T22:28:57.908 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:28:58.910 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:28:58.910 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:28:58.911 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:28:58.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:28:58.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:28:58.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:28:59.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:00.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:00.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:00.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:29:00.151 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:29:00.151 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:00.152 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:00.392 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:00.429 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:00.425+0000 7f97b330d7c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:29:01.393 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:01.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:01.394 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:29:01.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:29:01.394 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:01.395 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:01.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:02.826 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:02.826 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:02.826 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:29:02.827 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:29:02.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:02.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:03.067 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:04.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:04.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:04.069 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:29:04.070 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:29:04.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:04.070 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:04.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:05.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:05.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:05.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:29:05.311 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:29:05.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:05.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:05.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:06.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:06.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:06.554 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:29:06.555 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:29:06.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:06.555 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:06.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:07.798 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:07.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:07.799 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:29:07.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:29:07.799 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:07.800 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:08.041 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:09.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:09.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:09.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:29:09.043 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:29:09.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:09.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:09.285 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:10.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:10.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:10.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:29:10.287 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:29:10.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:10.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:10.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:11.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:11.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:11.530 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:29:11.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:29:11.530 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:11.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:11.773 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:12.775 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:12.776 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:12.776 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:29:12.776 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:29:12.776 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:12.776 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:13.017 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:14.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:14.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:14.019 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:29:14.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:29:14.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:14.020 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:14.260 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:15.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:15.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:15.262 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:29:15.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:29:15.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:15.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:15.499 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:16.501 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:16.502 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:16.502 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:29:16.502 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:29:16.502 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:16.503 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:16.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:17.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:17.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:17.742 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:29:17.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:29:17.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:17.742 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:17.974 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:18.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:18.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:18.976 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:29:18.977 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:29:18.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:18.977 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:19.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:20.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:20.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:20.218 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:29:20.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:29:20.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:20.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:20.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:21.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:21.461 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:21.462 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T22:29:21.462 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T22:29:21.462 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:21.462 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:21.699 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:22.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:22.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:22.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T22:29:22.702 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T22:29:22.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:22.702 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:22.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:23.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:23.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:23.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T22:29:23.945 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T22:29:23.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:23.945 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:24.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:25.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:25.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:25.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-27T22:29:25.187 INFO:tasks.workunit.client.0.smithi042.stdout:22 2024-06-27T22:29:25.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:25.187 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:29:25.439 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/374288518,v1:127.0.0.1:6801/374288518] [v2:127.0.0.1:6802/374288518,v1:127.0.0.1:6803/374288518] exists,up 011120a2-2b59-46a3-bdf1-39052865b83b 2024-06-27T22:29:25.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:29:25.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:29:25.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:29:25.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:79: TEST_check_version_health_1: run_osd td/ver-health 1 2024-06-27T22:29:25.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-27T22:29:25.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:29:25.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:29:25.441 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:29:25.442 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:29:25.443 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:29:25.443 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:29:25.443 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:29:25.443 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:29:25.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:29:25.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:29:25.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:29:25.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:29:25.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-27T22:29:25.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-27T22:29:25.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:29:25.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:29:25.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:29:25.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:29:25.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:29:25.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:29:25.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-06-27T22:29:25.445 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:29:25.445 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ebc5b409-2e18-40e2-8bef-7f04f9b55a0f 2024-06-27T22:29:25.445 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 ebc5b409-2e18-40e2-8bef-7f04f9b55a0f 2024-06-27T22:29:25.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ebc5b409-2e18-40e2-8bef-7f04f9b55a0f' 2024-06-27T22:29:25.446 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:29:25.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDF531mUzbRGxAAXtzfa3JRBBLgr4HsnPSovg== 2024-06-27T22:29:25.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDF531mUzbRGxAAXtzfa3JRBBLgr4HsnPSovg=="}' 2024-06-27T22:29:25.466 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ebc5b409-2e18-40e2-8bef-7f04f9b55a0f -i td/ver-health/1/new.json 2024-06-27T22:29:26.056 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:29:26.079 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-06-27T22:29:26.081 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDF531mUzbRGxAAXtzfa3JRBBLgr4HsnPSovg== --osd-uuid ebc5b409-2e18-40e2-8bef-7f04f9b55a0f 2024-06-27T22:29:26.117 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:26.113+0000 7f70774b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:29:26.120 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:26.117+0000 7f70774b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:29:26.121 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:26.117+0000 7f70774b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:29:26.122 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:26.117+0000 7f70774b17c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-27T22:29:26.122 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:26.117+0000 7f70774b17c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-27T22:29:26.122 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:26.117+0000 7f70774b17c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-27T22:29:26.239 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:26.233+0000 7f70774b17c0 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-06-27T22:29:32.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-06-27T22:29:32.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:29:32.855 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:29:32.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:29:32.856 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:29:33.110 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:29:33.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:29:33.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:29:33.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:29:33.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:29:33.111 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:29:33.143 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:33.137+0000 7fccf83697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:29:33.145 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:33.141+0000 7fccf83697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:29:33.146 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:33.141+0000 7fccf83697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:29:33.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:29:33.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:29:33.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:29:33.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:29:33.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:29:33.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:33.390 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:29:33.391 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:29:33.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:33.391 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:33.688 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:33.721 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:33.717+0000 7fccf83697c0 -1 Falling back to public interface 2024-06-27T22:29:34.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:34.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:34.690 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:29:34.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:29:34.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:34.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:34.928 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:35.930 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:35.930 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:35.930 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:29:35.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:29:35.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:35.931 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:36.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:36.297 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:29:36.293+0000 7fccf83697c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:29:37.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:37.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:37.173 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:29:37.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:29:37.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:37.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:37.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:38.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:38.533 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:38.533 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:29:38.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:29:38.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:38.534 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:38.776 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:39.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:39.778 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:39.779 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:29:39.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:29:39.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:39.779 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:40.019 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:41.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:41.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:41.021 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:29:41.022 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:29:41.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:41.022 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:41.262 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:42.264 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:42.264 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:42.264 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:29:42.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:29:42.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:42.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:42.508 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:43.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:43.510 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:43.510 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:29:43.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:29:43.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:43.511 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:43.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:44.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:44.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:44.752 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:29:44.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:29:44.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:44.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:44.998 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:46.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:46.000 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:46.000 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:29:46.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:29:46.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:46.001 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:46.240 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:47.241 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:47.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:47.242 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:29:47.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:29:47.242 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:47.243 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:47.481 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:48.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:48.483 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:48.483 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:29:48.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:29:48.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:48.484 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:48.726 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:49.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:49.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:49.728 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:29:49.728 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:29:49.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:49.729 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:49.967 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:50.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:50.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:50.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:29:50.970 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:29:50.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:50.970 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:51.211 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:52.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:52.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:52.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:29:52.214 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:29:52.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:52.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:52.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:53.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:53.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:53.458 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:29:53.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:29:53.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:53.459 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:53.697 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:54.699 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:54.700 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:54.700 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:29:54.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:29:54.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:54.701 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:54.942 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:55.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:55.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:55.943 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:29:55.943 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:29:55.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:55.944 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:56.183 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:57.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:57.185 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:57.185 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T22:29:57.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T22:29:57.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:57.186 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:57.424 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:58.426 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:58.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:58.429 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T22:29:58.429 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T22:29:58.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:58.430 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:58.666 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:29:59.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:29:59.668 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:29:59.669 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T22:29:59.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T22:29:59.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:29:59.670 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:29:59.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:00.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:00.911 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:00.912 INFO:tasks.workunit.client.0.smithi042.stdout:22 2024-06-27T22:30:00.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-27T22:30:00.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:00.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:30:01.207 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/608097331,v1:127.0.0.1:6809/608097331] [v2:127.0.0.1:6810/608097331,v1:127.0.0.1:6811/608097331] exists,up ebc5b409-2e18-40e2-8bef-7f04f9b55a0f 2024-06-27T22:30:01.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:30:01.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:30:01.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:30:01.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:80: TEST_check_version_health_1: run_osd td/ver-health 2 2024-06-27T22:30:01.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-27T22:30:01.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:30:01.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-27T22:30:01.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:30:01.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-06-27T22:30:01.214 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-27T22:30:01.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:30:01.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:30:01.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:30:01.215 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-06-27T22:30:01.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-06-27T22:30:01.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:30:01.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:30:01.216 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-27T22:30:01.216 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:30:01.216 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:30:01.217 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:30:01.217 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:30:01.217 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:30:01.217 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:30:01.217 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:30:01.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:30:01.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:30:01.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:30:01.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:30:01.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-27T22:30:01.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-27T22:30:01.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:30:01.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:30:01.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:30:01.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:30:01.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:30:01.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:30:01.220 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-06-27T22:30:01.220 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:30:01.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=72f699dc-0a87-4e13-a666-2739796f84a3 2024-06-27T22:30:01.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 72f699dc-0a87-4e13-a666-2739796f84a3' 2024-06-27T22:30:01.222 INFO:tasks.workunit.client.0.smithi042.stdout:add osd2 72f699dc-0a87-4e13-a666-2739796f84a3 2024-06-27T22:30:01.222 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:30:01.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDp531mfPbbDRAAFbqJFqFX2mTx/SkjwwwM6g== 2024-06-27T22:30:01.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDp531mfPbbDRAAFbqJFqFX2mTx/SkjwwwM6g=="}' 2024-06-27T22:30:01.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 72f699dc-0a87-4e13-a666-2739796f84a3 -i td/ver-health/2/new.json 2024-06-27T22:30:02.320 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:30:02.346 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-06-27T22:30:02.347 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDp531mfPbbDRAAFbqJFqFX2mTx/SkjwwwM6g== --osd-uuid 72f699dc-0a87-4e13-a666-2739796f84a3 2024-06-27T22:30:02.385 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:02.380+0000 7f61b5c637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:02.409 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:02.404+0000 7f61b5c637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:02.411 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:02.408+0000 7f61b5c637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:02.411 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:02.408+0000 7f61b5c637c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-27T22:30:02.412 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:02.408+0000 7f61b5c637c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-27T22:30:02.412 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:02.408+0000 7f61b5c637c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-27T22:30:02.527 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:02.520+0000 7f61b5c637c0 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-06-27T22:30:08.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-06-27T22:30:08.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:30:08.677 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd2 key to auth repository 2024-06-27T22:30:08.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-27T22:30:08.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:30:08.934 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T22:30:08.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-27T22:30:08.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:30:08.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:30:08.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:30:08.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:30:08.968 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:08.964+0000 7f457be647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:08.970 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:08.964+0000 7f457be647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:08.971 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:08.968+0000 7f457be647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:09.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-27T22:30:09.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:30:09.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T22:30:09.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:30:09.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:30:09.218 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:09.218 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:30:09.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:30:09.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:09.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:09.505 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:09.500+0000 7f457be647c0 -1 Falling back to public interface 2024-06-27T22:30:09.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:10.550 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:10.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:10.551 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:30:10.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:30:10.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:10.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:10.791 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:11.792 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:11.793 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:11.793 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:30:11.793 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:30:11.793 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:11.793 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:12.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:12.180 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:12.176+0000 7f457be647c0 -1 osd.2 0 log_to_monitors true 2024-06-27T22:30:13.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:13.033 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:13.033 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:30:13.034 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:30:13.034 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:13.034 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:13.492 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:14.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:14.494 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:14.495 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:30:14.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:30:14.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:14.495 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:14.735 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:15.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:15.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:15.737 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:30:15.737 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:30:15.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:15.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:15.978 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:16.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:16.980 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:16.981 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:30:16.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:30:16.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:16.981 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:17.219 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:18.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:18.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:18.222 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:30:18.222 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:30:18.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:18.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:18.465 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:19.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:19.467 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:19.467 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:30:19.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:30:19.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:19.468 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:19.706 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:20.707 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:20.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:20.708 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:30:20.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:30:20.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:20.709 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:20.948 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:21.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:21.949 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:21.950 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:30:21.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:30:21.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:21.950 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:22.189 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:23.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:23.191 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:23.191 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:30:23.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:30:23.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:23.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:23.431 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:24.433 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:24.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:24.434 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:30:24.434 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:30:24.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:24.435 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:24.674 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:25.675 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:25.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:25.676 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:30:25.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:30:25.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:25.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:25.919 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:26.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:26.920 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:26.921 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:30:26.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:30:26.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:26.921 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:27.158 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:28.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:28.159 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:28.160 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:30:28.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:30:28.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:28.160 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:28.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:29.403 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:29.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:29.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:30:29.404 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:30:29.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:29.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:29.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:30.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:30.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:30.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:30:30.648 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:30:30.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:30.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:30.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:31.890 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:31.890 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:31.891 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:30:31.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:30:31.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:31.891 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:32.128 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:33.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:33.131 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T22:30:33.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:33.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T22:30:33.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:33.131 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:33.372 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:34.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:34.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:34.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T22:30:34.374 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T22:30:34.374 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:34.375 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:34.616 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:35.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:35.618 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:35.618 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T22:30:35.619 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T22:30:35.619 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:35.619 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:35.859 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:36.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:36.861 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:36.861 INFO:tasks.workunit.client.0.smithi042.stdout:22 2024-06-27T22:30:36.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-27T22:30:36.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:36.862 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:37.611 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/476101374,v1:127.0.0.1:6817/476101374] [v2:127.0.0.1:6818/476101374,v1:127.0.0.1:6819/476101374] exists,up 72f699dc-0a87-4e13-a666-2739796f84a3 2024-06-27T22:30:37.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:30:37.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:30:37.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:30:37.612 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:82: TEST_check_version_health_1: sleep 5 2024-06-27T22:30:42.614 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:83: TEST_check_version_health_1: ceph health detail 2024-06-27T22:30:42.844 INFO:tasks.workunit.client.0.smithi042.stdout:HEALTH_OK 2024-06-27T22:30:42.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: ceph health detail 2024-06-27T22:30:42.860 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: grep DAEMON_OLD_VERSION 2024-06-27T22:30:43.221 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:87: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.1 2024-06-27T22:30:43.222 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:30:43.222 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:30:43.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:30:43.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:30:43.223 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:30:43.331 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:30:43.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-06-27T22:30:43.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: activate_osd td/ver-health 1 2024-06-27T22:30:43.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-06-27T22:30:43.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-27T22:30:43.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-27T22:30:43.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-27T22:30:43.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2024-06-27T22:30:43.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-27T22:30:43.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:30:43.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:30:43.332 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:30:43.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-06-27T22:30:43.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-06-27T22:30:43.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-27T22:30:43.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-27T22:30:43.333 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-27T22:30:43.335 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-27T22:30:43.335 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:30:43.335 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:30:43.335 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:30:43.335 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:30:43.335 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:30:43.335 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:30:43.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:30:43.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:30:43.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-27T22:30:43.335 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-27T22:30:43.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:30:43.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:30:43.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:30:43.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:30:43.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-27T22:30:43.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-27T22:30:43.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2024-06-27T22:30:43.336 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:30:43.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-27T22:30:43.337 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:30:43.337 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2024-06-27T22:30:43.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-27T22:30:43.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-27T22:30:43.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-27T22:30:43.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-27T22:30:43.371 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:43.368+0000 7f6a650987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:43.380 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:43.376+0000 7f6a650987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:43.382 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:43.376+0000 7f6a650987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:43.918 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:43.912+0000 7f6a650987c0 -1 Falling back to public interface 2024-06-27T22:30:44.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-27T22:30:44.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:30:44.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:30:44.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:30:44.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:30:44.046 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:30:44.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:44.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:30:44.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:44.046 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:30:44.291 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:45.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:45.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:45.293 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:30:45.293 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:30:45.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:45.294 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:30:45.538 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:46.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:46.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:46.540 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:30:46.540 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:30:46.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:46.541 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:30:46.781 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:46.789 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:46.784+0000 7f6a650987c0 -1 osd.1 16 log_to_monitors true 2024-06-27T22:30:47.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:47.782 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:47.783 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:30:47.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:30:47.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:47.783 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:30:48.573 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6808/3369020764,v1:127.0.0.1:6809/3369020764] [v2:127.0.0.1:6810/3369020764,v1:127.0.0.1:6811/3369020764] exists,up ebc5b409-2e18-40e2-8bef-7f04f9b55a0f 2024-06-27T22:30:48.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:30:48.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:30:48.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:30:48.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:90: TEST_check_version_health_1: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-27T22:30:48.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-27T22:30:48.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-06-27T22:30:48.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-06-27T22:30:48.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-06-27T22:30:48.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-06-27T22:30:48.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-27T22:30:48.575 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-27T22:30:48.576 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-27T22:30:48.844 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-27T22:30:49.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-27T22:30:49.846 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-27T22:30:49.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-27T22:30:49.847 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-27T22:30:50.138 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-27T22:30:51.139 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-27T22:30:51.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-27T22:30:51.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-27T22:30:51.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-27T22:30:51.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-27T22:30:52.388 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-27T22:30:52.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-27T22:30:52.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-27T22:30:52.389 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-27T22:30:52.634 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-27T22:30:53.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-27T22:30:53.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-27T22:30:53.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-27T22:30:53.636 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-27T22:30:53.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-06-27T22:30:53.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-06-27T22:30:53.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-06-27T22:30:53.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-06-27T22:30:53.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-06-27T22:30:53.886 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:92: TEST_check_version_health_1: ceph health detail 2024-06-27T22:30:54.176 INFO:tasks.workunit.client.0.smithi042.stdout:HEALTH_WARN There is a daemon running an older version of ceph; no active mgr 2024-06-27T22:30:54.176 INFO:tasks.workunit.client.0.smithi042.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2024-06-27T22:30:54.176 INFO:tasks.workunit.client.0.smithi042.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2024-06-27T22:30:54.176 INFO:tasks.workunit.client.0.smithi042.stdout:[WRN] MGR_DOWN: no active mgr 2024-06-27T22:30:54.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: ceph health detail 2024-06-27T22:30:54.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-06-27T22:30:54.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: ceph health detail 2024-06-27T22:30:54.458 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2024-06-27T22:30:54.795 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: ceph health detail 2024-06-27T22:30:54.795 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2024-06-27T22:30:55.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:98: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.2 2024-06-27T22:30:55.168 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:30:55.168 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:30:55.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:30:55.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:30:55.168 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:30:55.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:30:55.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-06-27T22:30:55.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: activate_osd td/ver-health 2 2024-06-27T22:30:55.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-06-27T22:30:55.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-27T22:30:55.279 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:30:55.280 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:30:55.280 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:30:55.280 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:30:55.280 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:30:55.280 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:30:55.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:30:55.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:30:55.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-27T22:30:55.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-27T22:30:55.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:30:55.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:30:55.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:30:55.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:30:55.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-27T22:30:55.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-27T22:30:55.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-06-27T22:30:55.282 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-27T22:30:55.282 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T22:30:55.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:30:55.284 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-06-27T22:30:55.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-27T22:30:55.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-27T22:30:55.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-27T22:30:55.284 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-27T22:30:55.314 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:55.308+0000 7fb516eee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:55.362 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:55.356+0000 7fb516eee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:55.364 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:55.360+0000 7fb516eee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:30:55.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-27T22:30:55.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:30:55.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T22:30:55.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:30:55.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:30:55.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:55.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:30:55.605 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:30:55.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:55.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:55.918 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:55.912+0000 7fb516eee7c0 -1 Falling back to public interface 2024-06-27T22:30:56.040 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:57.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:57.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:57.043 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:30:57.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:30:57.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:57.043 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:57.278 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:58.279 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:58.280 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:30:58.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:58.280 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:30:58.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:58.281 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:30:58.759 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:30:59.242 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:30:59.240+0000 7fb516eee7c0 -1 osd.2 19 log_to_monitors true 2024-06-27T22:30:59.761 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:30:59.761 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:30:59.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:30:59.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:30:59.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:30:59.762 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:31:00.065 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 20 last_clean_interval [15,19) [v2:127.0.0.1:6816/1474037823,v1:127.0.0.1:6817/1474037823] [v2:127.0.0.1:6818/1474037823,v1:127.0.0.1:6819/1474037823] exists,up 72f699dc-0a87-4e13-a666-2739796f84a3 2024-06-27T22:31:00.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:31:00.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:31:00.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:31:00.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:100: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.0 2024-06-27T22:31:00.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:31:00.066 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:31:00.067 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:31:00.067 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:31:00.067 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:31:00.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:31:00.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-06-27T22:31:00.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: activate_osd td/ver-health 0 2024-06-27T22:31:00.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-06-27T22:31:00.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-27T22:31:00.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-27T22:31:00.175 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-27T22:31:00.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-06-27T22:31:00.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-27T22:31:00.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:31:00.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:31:00.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:31:00.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-06-27T22:31:00.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-06-27T22:31:00.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-27T22:31:00.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-27T22:31:00.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-27T22:31:00.178 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-27T22:31:00.178 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:31:00.178 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:31:00.178 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:31:00.178 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:00.178 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:00.178 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:31:00.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:31:00.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:31:00.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-27T22:31:00.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-27T22:31:00.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:31:00.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:31:00.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:31:00.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:31:00.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-27T22:31:00.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-27T22:31:00.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-06-27T22:31:00.179 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:31:00.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-27T22:31:00.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:31:00.180 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-06-27T22:31:00.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-27T22:31:00.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-27T22:31:00.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-27T22:31:00.181 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-27T22:31:00.212 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:00.208+0000 7fc56c63d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:00.215 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:00.212+0000 7fc56c63d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:00.217 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:00.212+0000 7fc56c63d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:00.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-27T22:31:00.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:31:00.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:31:00.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:31:00.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:31:00.604 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:00.604 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:31:00.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:31:00.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:00.605 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:00.780 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:00.776+0000 7fc56c63d7c0 -1 Falling back to public interface 2024-06-27T22:31:01.042 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:02.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:02.044 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:02.045 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:31:02.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:31:02.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:02.045 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:02.286 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:03.287 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:03.288 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:03.288 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:31:03.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:31:03.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:03.289 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:03.529 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:04.094 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:04.088+0000 7fc56c63d7c0 -1 osd.0 22 log_to_monitors true 2024-06-27T22:31:04.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:04.531 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:04.531 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:31:04.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:31:04.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:04.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:05.028 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6800/2964046398,v1:127.0.0.1:6801/2964046398] [v2:127.0.0.1:6802/2964046398,v1:127.0.0.1:6803/2964046398] exists,up 011120a2-2b59-46a3-bdf1-39052865b83b 2024-06-27T22:31:05.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:31:05.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:31:05.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:31:05.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:103: TEST_check_version_health_1: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-27T22:31:05.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-27T22:31:05.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-06-27T22:31:05.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-06-27T22:31:05.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-06-27T22:31:05.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-06-27T22:31:05.030 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-27T22:31:05.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-27T22:31:05.031 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-27T22:31:05.336 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-27T22:31:06.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-27T22:31:06.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-27T22:31:06.338 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-27T22:31:06.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-27T22:31:06.584 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-27T22:31:07.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-27T22:31:07.585 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-27T22:31:07.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-27T22:31:07.586 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-27T22:31:07.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-27T22:31:08.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-27T22:31:08.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-27T22:31:08.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-27T22:31:08.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-27T22:31:09.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-06-27T22:31:09.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-06-27T22:31:09.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-06-27T22:31:09.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-06-27T22:31:09.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-06-27T22:31:09.076 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:105: TEST_check_version_health_1: ceph health detail 2024-06-27T22:31:09.391 INFO:tasks.workunit.client.0.smithi042.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph; no active mgr 2024-06-27T22:31:09.391 INFO:tasks.workunit.client.0.smithi042.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-06-27T22:31:09.391 INFO:tasks.workunit.client.0.smithi042.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2024-06-27T22:31:09.391 INFO:tasks.workunit.client.0.smithi042.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-06-27T22:31:09.391 INFO:tasks.workunit.client.0.smithi042.stdout:[WRN] MGR_DOWN: no active mgr 2024-06-27T22:31:09.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: ceph health detail 2024-06-27T22:31:09.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-06-27T22:31:09.744 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-06-27T22:31:09.745 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: ceph health detail 2024-06-27T22:31:10.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: ceph health detail 2024-06-27T22:31:10.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2024-06-27T22:31:10.384 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-06-27T22:31:10.402 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: ceph health detail 2024-06-27T22:31:10.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-06-27T22:31:10.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-27T22:31:10.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:31:10.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-27T22:31:10.676 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:31:10.676 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:31:10.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:31:10.676 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:31:10.677 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:31:10.785 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:31:10.786 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:31:10.786 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:31:10.786 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:31:10.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:31:10.787 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:31:10.787 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:31:10.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:31:10.788 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:31:10.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:31:10.789 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:31:10.789 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:31:10.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:31:10.790 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-27T22:31:10.808 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:31:10.809 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:10.809 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:10.809 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.183441 2024-06-27T22:31:10.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:31:10.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:31:10.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-06-27T22:31:10.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-06-27T22:31:10.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-06-27T22:31:10.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-06-27T22:31:10.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-27T22:31:10.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:31:10.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-27T22:31:10.812 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:31:10.812 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:31:10.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:31:10.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:31:10.812 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:31:10.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:31:10.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:31:10.813 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:31:10.813 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:31:10.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:31:10.814 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:31:10.814 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:31:10.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:31:10.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:31:10.815 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:31:10.816 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:31:10.816 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:31:10.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:31:10.817 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-27T22:31:10.818 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:31:10.818 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:10.819 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:10.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.183441 2024-06-27T22:31:10.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:31:10.819 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:31:10.820 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-06-27T22:31:10.820 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:31:10.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:10.821 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:10.821 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.183441 2024-06-27T22:31:10.822 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_2 td/ver-health 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:114: TEST_check_version_health_2: local dir=td/ver-health 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:117: TEST_check_version_health_2: CEPH_ARGS='--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:119: TEST_check_version_health_2: setup td/ver-health 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-27T22:31:10.824 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-27T22:31:10.825 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:31:10.825 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:31:10.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:31:10.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:31:10.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:31:10.825 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:31:10.825 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:31:10.826 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:31:10.826 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:31:10.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:31:10.827 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:31:10.828 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:31:10.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:31:10.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:31:10.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:31:10.829 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:31:10.829 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:31:10.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-27T22:31:10.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-27T22:31:10.832 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:31:10.832 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:10.832 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:10.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.183441 2024-06-27T22:31:10.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:31:10.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:31:10.833 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-06-27T22:31:10.834 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-27T22:31:10.834 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:10.834 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:10.834 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.183441 2024-06-27T22:31:10.836 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-27T22:31:10.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-27T22:31:10.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-27T22:31:10.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-06-27T22:31:10.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:122: TEST_check_version_health_2: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-27T22:31:10.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-06-27T22:31:10.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:31:10.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-27T22:31:10.836 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:31:10.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-06-27T22:31:10.837 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-27T22:31:11.317 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:31:11.317 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:31:11.317 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:31:11.318 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:31:11.318 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:11.318 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:11.318 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:31:11.319 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-06-27T22:31:11.528 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:31:11.530 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-27T22:31:11.530 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:31:11.530 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:31:11.530 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:31:11.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:31:11.531 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:31:11.531 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:31:11.531 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:31:11.532 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:31:11.532 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:11.532 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:11.532 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.183441/ceph-mon.a.asok 2024-06-27T22:31:11.532 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:31:11.532 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.183441/ceph-mon.a.asok config get fsid 2024-06-27T22:31:11.649 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-27T22:31:11.649 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:31:11.649 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-27T22:31:11.650 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:31:11.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:31:11.651 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-27T22:31:11.651 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-27T22:31:11.651 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-27T22:31:11.651 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:31:11.651 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:11.651 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:11.651 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.183441/ceph-mon.a.asok 2024-06-27T22:31:11.652 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:31:11.652 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.183441/ceph-mon.a.asok config get mon_host 2024-06-27T22:31:11.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:123: TEST_check_version_health_2: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-27T22:31:11.763 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-06-27T22:31:11.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:31:11.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-27T22:31:11.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:31:11.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-06-27T22:31:11.764 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-27T22:31:12.706 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:31:12.706 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:31:12.706 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:31:12.707 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:31:12.707 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:12.707 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:12.707 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:31:12.708 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-06-27T22:31:12.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:31:12.910 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-27T22:31:12.910 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:31:12.910 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T22:31:12.910 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:31:12.911 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:31:12.911 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T22:31:12.911 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T22:31:12.912 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T22:31:12.912 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:31:12.912 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:12.912 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:12.912 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.183441/ceph-mon.b.asok 2024-06-27T22:31:12.912 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:31:12.912 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.183441/ceph-mon.b.asok config get fsid 2024-06-27T22:31:13.030 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-27T22:31:13.030 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:31:13.030 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T22:31:13.030 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:31:13.031 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:31:13.031 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T22:31:13.031 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T22:31:13.032 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T22:31:13.032 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:31:13.032 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:13.032 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:13.032 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.183441/ceph-mon.b.asok 2024-06-27T22:31:13.032 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:31:13.032 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.183441/ceph-mon.b.asok config get mon_host 2024-06-27T22:31:13.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:124: TEST_check_version_health_2: run_osd td/ver-health 0 2024-06-27T22:31:13.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-27T22:31:13.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:31:13.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-27T22:31:13.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:31:13.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:31:13.144 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:31:13.145 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:31:13.145 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:13.145 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:31:13.146 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:31:13.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-06-27T22:31:13.147 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:31:13.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=22d3d8fd-ae21-4450-8d56-aeeab871564f 2024-06-27T22:31:13.147 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 22d3d8fd-ae21-4450-8d56-aeeab871564f' 2024-06-27T22:31:13.148 INFO:tasks.workunit.client.0.smithi042.stdout:add osd0 22d3d8fd-ae21-4450-8d56-aeeab871564f 2024-06-27T22:31:13.148 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:31:13.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAx6H1m8hUlChAAB8jU9uW9Mi8qmrVO8+ISAg== 2024-06-27T22:31:13.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAx6H1m8hUlChAAB8jU9uW9Mi8qmrVO8+ISAg=="}' 2024-06-27T22:31:13.170 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 22d3d8fd-ae21-4450-8d56-aeeab871564f -i td/ver-health/0/new.json 2024-06-27T22:31:19.549 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:31:19.568 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-06-27T22:31:19.570 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAx6H1m8hUlChAAB8jU9uW9Mi8qmrVO8+ISAg== --osd-uuid 22d3d8fd-ae21-4450-8d56-aeeab871564f 2024-06-27T22:31:19.608 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:19.608+0000 7f5faf8817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:19.686 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:19.684+0000 7f5faf8817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:19.688 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:19.688+0000 7f5faf8817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:19.688 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:19.688+0000 7f5faf8817c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-27T22:31:19.689 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:19.688+0000 7f5faf8817c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-27T22:31:19.689 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:19.688+0000 7f5faf8817c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-06-27T22:31:19.777 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:19.776+0000 7f5faf8817c0 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-06-27T22:31:25.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-06-27T22:31:25.924 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:31:25.926 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd0 key to auth repository 2024-06-27T22:31:25.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-27T22:31:25.926 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:31:26.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-27T22:31:26.179 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.0 2024-06-27T22:31:26.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:31:26.179 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:31:26.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:31:26.180 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:31:26.211 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:26.212+0000 7f9af22387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:26.213 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:26.212+0000 7f9af22387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:26.215 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:26.216+0000 7f9af22387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:26.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-27T22:31:26.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:31:26.469 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-27T22:31:26.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:31:26.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:31:26.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:26.470 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:31:26.470 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:31:26.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:26.471 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:26.749 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:26.748+0000 7f9af22387c0 -1 Falling back to public interface 2024-06-27T22:31:26.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:27.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:27.810 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:27.810 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:31:27.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:31:27.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:27.811 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:28.051 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:29.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:29.053 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:29.054 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:31:29.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:31:29.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:29.054 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:29.297 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:29.331 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:29.328+0000 7f9af22387c0 -1 osd.0 0 log_to_monitors true 2024-06-27T22:31:30.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:30.299 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:30.299 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:31:30.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:31:30.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:30.300 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:30.969 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:31.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:31.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:31.971 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:31:31.971 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:31:31.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:31.972 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:32.210 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:33.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:33.212 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:33.212 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:31:33.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:31:33.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:33.213 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:33.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:34.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:34.455 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:34.455 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:31:34.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:31:34.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:34.456 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:34.693 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:35.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:35.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:35.695 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:31:35.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:31:35.695 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:35.696 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:35.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:36.933 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:36.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:36.934 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:31:36.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:31:36.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:36.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:37.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:38.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:38.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:38.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:31:38.178 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:31:38.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:38.178 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:38.420 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:39.421 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:39.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:39.422 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:31:39.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:31:39.422 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:39.423 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:39.660 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:40.663 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:40.663 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:40.663 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:31:40.663 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:31:40.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:40.664 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:40.899 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:41.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:41.901 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:41.902 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:31:41.902 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:31:41.902 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:41.902 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:42.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:43.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:43.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:43.142 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:31:43.142 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:31:43.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:43.143 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:43.384 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:44.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:44.386 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:44.387 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:31:44.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:31:44.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:44.387 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:44.628 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:45.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:45.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:45.630 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:31:45.630 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:31:45.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:45.631 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:45.869 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:46.871 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:46.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:46.872 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:31:46.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:31:46.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:46.872 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:47.110 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:48.112 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:48.112 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:48.112 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:31:48.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:31:48.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:48.113 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:48.404 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:49.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:49.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:49.406 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:31:49.406 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:31:49.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:49.407 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:49.643 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:50.645 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:50.646 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:50.646 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T22:31:50.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T22:31:50.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:50.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:50.885 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:51.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:51.887 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:51.888 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T22:31:51.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T22:31:51.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:51.888 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:52.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:53.124 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:53.124 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:53.124 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T22:31:53.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T22:31:53.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:53.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:53.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:31:54.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:31:54.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:31:54.366 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-27T22:31:54.366 INFO:tasks.workunit.client.0.smithi042.stdout:22 2024-06-27T22:31:54.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:31:54.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-27T22:31:54.643 INFO:tasks.workunit.client.0.smithi042.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3734679490,v1:127.0.0.1:6801/3734679490] [v2:127.0.0.1:6802/3734679490,v1:127.0.0.1:6803/3734679490] exists,up 22d3d8fd-ae21-4450-8d56-aeeab871564f 2024-06-27T22:31:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:31:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:31:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:31:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:125: TEST_check_version_health_2: run_osd td/ver-health 1 2024-06-27T22:31:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-27T22:31:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:31:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-27T22:31:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:31:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-06-27T22:31:54.647 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:31:54.648 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:31:54.649 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:31:54.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-06-27T22:31:54.650 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:31:54.650 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=253b8492-6c5c-4750-98cc-9643a979b4d4 2024-06-27T22:31:54.650 INFO:tasks.workunit.client.0.smithi042.stdout:add osd1 253b8492-6c5c-4750-98cc-9643a979b4d4 2024-06-27T22:31:54.651 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 253b8492-6c5c-4750-98cc-9643a979b4d4' 2024-06-27T22:31:54.651 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:31:54.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBa6H1m4w3gJxAAH4EVvCDu7mEfCOSx3nu28A== 2024-06-27T22:31:54.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBa6H1m4w3gJxAAH4EVvCDu7mEfCOSx3nu28A=="}' 2024-06-27T22:31:54.669 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 253b8492-6c5c-4750-98cc-9643a979b4d4 -i td/ver-health/1/new.json 2024-06-27T22:31:55.141 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:31:55.162 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-06-27T22:31:55.164 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBa6H1m4w3gJxAAH4EVvCDu7mEfCOSx3nu28A== --osd-uuid 253b8492-6c5c-4750-98cc-9643a979b4d4 2024-06-27T22:31:55.201 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:55.200+0000 7f38a99397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:55.204 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:55.204+0000 7f38a99397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:55.205 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:55.204+0000 7f38a99397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:31:55.206 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:55.204+0000 7f38a99397c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-27T22:31:55.206 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:55.204+0000 7f38a99397c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-27T22:31:55.206 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:55.204+0000 7f38a99397c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-06-27T22:31:55.317 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:31:55.316+0000 7f38a99397c0 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-06-27T22:32:01.912 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-06-27T22:32:01.913 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:32:01.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-27T22:32:01.914 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd1 key to auth repository 2024-06-27T22:32:01.914 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:32:02.170 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.1 2024-06-27T22:32:02.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-27T22:32:02.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:32:02.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:32:02.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:32:02.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:32:02.203 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:02.204+0000 7f5a9eb8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:32:02.206 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:02.204+0000 7f5a9eb8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:32:02.207 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:02.208+0000 7f5a9eb8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:32:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-27T22:32:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:32:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-27T22:32:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:32:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:32:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:02.474 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:32:02.474 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:32:02.474 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:02.474 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:02.748 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:02.785 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:02.784+0000 7f5a9eb8b7c0 -1 Falling back to public interface 2024-06-27T22:32:03.750 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:03.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:03.751 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:32:03.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:32:03.751 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:03.752 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:03.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:04.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:04.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:04.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:32:04.992 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:32:04.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:04.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:05.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:05.440 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:05.440+0000 7f5a9eb8b7c0 -1 osd.1 0 log_to_monitors true 2024-06-27T22:32:06.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:06.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:06.234 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:32:06.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:32:06.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:06.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:06.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:07.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:07.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:07.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:32:07.475 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:32:07.476 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:07.476 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:07.716 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:08.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:08.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:08.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:32:08.719 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:32:08.719 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:08.719 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:09.096 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:10.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:10.098 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:10.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:32:10.099 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:32:10.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:10.100 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:10.339 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:11.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:11.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:11.341 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:32:11.341 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:32:11.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:11.342 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:11.581 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:12.583 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:12.830 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:12.831 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:32:12.831 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:32:12.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:12.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:12.832 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:13.822 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:13.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:13.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:32:13.823 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:32:13.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:13.823 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:14.064 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:15.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:15.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:15.066 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:32:15.066 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:32:15.067 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:15.067 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:15.308 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:16.310 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:16.310 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:32:16.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:32:16.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:16.311 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:16.549 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:17.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:17.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:17.551 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:32:17.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:32:17.552 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:17.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:17.794 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:18.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:18.796 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:18.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:32:18.797 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:32:18.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:18.797 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:19.261 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:20.264 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:20.264 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:20.264 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:32:20.265 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:32:20.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:20.265 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:21.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:22.155 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:22.155 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:22.155 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:32:22.155 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:32:22.156 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:22.156 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:22.398 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:23.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:23.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:23.400 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:32:23.400 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:32:23.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:23.401 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:23.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:24.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:24.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:24.640 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:32:24.640 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:32:24.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:24.641 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:24.881 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:25.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:25.882 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:25.882 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:32:25.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:32:25.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:25.883 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:26.123 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:27.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:27.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:27.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T22:32:27.126 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T22:32:27.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:27.126 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:27.365 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:28.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:28.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:28.367 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T22:32:28.368 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T22:32:28.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:28.369 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:28.606 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:29.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:29.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:29.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T22:32:29.608 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T22:32:29.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:29.609 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:29.906 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:30.908 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:30.908 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:30.908 INFO:tasks.workunit.client.0.smithi042.stdout:22 2024-06-27T22:32:30.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-27T22:32:30.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:30.909 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-27T22:32:31.147 INFO:tasks.workunit.client.0.smithi042.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/3714930184,v1:127.0.0.1:6809/3714930184] [v2:127.0.0.1:6810/3714930184,v1:127.0.0.1:6811/3714930184] exists,up 253b8492-6c5c-4750-98cc-9643a979b4d4 2024-06-27T22:32:31.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:32:31.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:32:31.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:32:31.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:126: TEST_check_version_health_2: run_osd td/ver-health 2 2024-06-27T22:32:31.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-06-27T22:32:31.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-27T22:32:31.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-27T22:32:31.149 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:32:31.150 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:32:31.152 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:32:31.152 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-27T22:32:31.153 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-27T22:32:31.154 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-06-27T22:32:31.154 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-27T22:32:31.154 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca3bab8b-59b0-4848-bc26-76728d8c1e01 2024-06-27T22:32:31.154 INFO:tasks.workunit.client.0.smithi042.stdout:add osd2 ca3bab8b-59b0-4848-bc26-76728d8c1e01 2024-06-27T22:32:31.155 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ca3bab8b-59b0-4848-bc26-76728d8c1e01' 2024-06-27T22:32:31.155 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-27T22:32:31.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/6H1mmZZJChAAs9IdcvREpQ+6IRA/yEKL/Q== 2024-06-27T22:32:31.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/6H1mmZZJChAAs9IdcvREpQ+6IRA/yEKL/Q=="}' 2024-06-27T22:32:31.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca3bab8b-59b0-4848-bc26-76728d8c1e01 -i td/ver-health/2/new.json 2024-06-27T22:32:32.256 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:32:32.283 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-06-27T22:32:32.285 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/6H1mmZZJChAAs9IdcvREpQ+6IRA/yEKL/Q== --osd-uuid ca3bab8b-59b0-4848-bc26-76728d8c1e01 2024-06-27T22:32:32.323 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:32.323+0000 7f9a429317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:32:32.325 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:32.323+0000 7f9a429317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:32:32.327 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:32.327+0000 7f9a429317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:32:32.328 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:32.327+0000 7f9a429317c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-27T22:32:32.328 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:32.327+0000 7f9a429317c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-27T22:32:32.328 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:32.327+0000 7f9a429317c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-06-27T22:32:32.438 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:32.435+0000 7f9a429317c0 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-06-27T22:32:38.805 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-06-27T22:32:38.806 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-27T22:32:38.807 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-27T22:32:38.807 INFO:tasks.workunit.client.0.smithi042.stdout:adding osd2 key to auth repository 2024-06-27T22:32:38.808 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-27T22:32:39.061 INFO:tasks.workunit.client.0.smithi042.stdout:start osd.2 2024-06-27T22:32:39.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-27T22:32:39.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9327a8ae-0e84-4a28-a27d-fbf5189a7581 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-27T22:32:39.061 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-27T22:32:39.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-27T22:32:39.062 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-27T22:32:39.095 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:39.091+0000 7f19ed59e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:32:39.097 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:39.095+0000 7f19ed59e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:32:39.100 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:39.099+0000 7f19ed59e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-27T22:32:39.356 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-27T22:32:39.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-27T22:32:39.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-27T22:32:39.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-27T22:32:39.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-27T22:32:39.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:39.357 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-27T22:32:39.357 INFO:tasks.workunit.client.0.smithi042.stdout:0 2024-06-27T22:32:39.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:39.358 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:39.635 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:39.657 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:39.655+0000 7f19ed59e7c0 -1 Falling back to public interface 2024-06-27T22:32:40.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:40.637 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:40.637 INFO:tasks.workunit.client.0.smithi042.stdout:1 2024-06-27T22:32:40.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-27T22:32:40.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:40.638 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:40.873 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:41.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:41.874 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:41.875 INFO:tasks.workunit.client.0.smithi042.stdout:2 2024-06-27T22:32:41.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-27T22:32:41.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:41.875 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:42.115 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:42.350 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:32:42.347+0000 7f19ed59e7c0 -1 osd.2 0 log_to_monitors true 2024-06-27T22:32:43.116 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:43.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:43.117 INFO:tasks.workunit.client.0.smithi042.stdout:3 2024-06-27T22:32:43.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-27T22:32:43.117 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:43.118 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:43.738 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:44.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:44.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:44.740 INFO:tasks.workunit.client.0.smithi042.stdout:4 2024-06-27T22:32:44.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-27T22:32:44.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:44.741 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:44.983 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:45.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:45.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:45.985 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-27T22:32:45.985 INFO:tasks.workunit.client.0.smithi042.stdout:5 2024-06-27T22:32:45.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:45.986 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:46.229 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:47.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:47.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:47.231 INFO:tasks.workunit.client.0.smithi042.stdout:6 2024-06-27T22:32:47.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-27T22:32:47.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:47.232 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:47.472 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:48.474 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:48.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:48.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-27T22:32:48.475 INFO:tasks.workunit.client.0.smithi042.stdout:7 2024-06-27T22:32:48.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:48.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:48.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:49.719 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:49.719 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:49.719 INFO:tasks.workunit.client.0.smithi042.stdout:8 2024-06-27T22:32:49.720 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-27T22:32:49.720 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:49.720 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:49.959 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:50.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:50.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:50.962 INFO:tasks.workunit.client.0.smithi042.stdout:9 2024-06-27T22:32:50.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-27T22:32:50.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:50.962 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:51.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:52.202 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:52.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:52.203 INFO:tasks.workunit.client.0.smithi042.stdout:10 2024-06-27T22:32:52.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-27T22:32:52.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:52.203 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:52.444 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:53.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:53.446 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:53.447 INFO:tasks.workunit.client.0.smithi042.stdout:11 2024-06-27T22:32:53.447 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-27T22:32:53.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:53.448 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:53.687 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:54.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:54.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:54.690 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-27T22:32:54.693 INFO:tasks.workunit.client.0.smithi042.stdout:12 2024-06-27T22:32:54.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:54.694 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:54.932 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:55.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:55.934 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:55.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-27T22:32:55.935 INFO:tasks.workunit.client.0.smithi042.stdout:13 2024-06-27T22:32:55.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:55.935 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:56.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:57.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:57.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:57.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-27T22:32:57.176 INFO:tasks.workunit.client.0.smithi042.stdout:14 2024-06-27T22:32:57.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:57.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:57.416 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:58.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:58.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:58.418 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-27T22:32:58.418 INFO:tasks.workunit.client.0.smithi042.stdout:15 2024-06-27T22:32:58.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:58.419 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:58.689 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:32:59.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:32:59.691 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:32:59.691 INFO:tasks.workunit.client.0.smithi042.stdout:16 2024-06-27T22:32:59.692 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-27T22:32:59.692 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:32:59.692 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:32:59.990 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:33:00.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:33:00.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:33:00.992 INFO:tasks.workunit.client.0.smithi042.stdout:17 2024-06-27T22:33:00.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-27T22:33:00.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:33:00.992 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:33:01.231 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:33:02.233 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:33:02.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:33:02.234 INFO:tasks.workunit.client.0.smithi042.stdout:18 2024-06-27T22:33:02.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-27T22:33:02.234 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:33:02.235 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:33:02.473 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:33:03.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:33:03.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:33:03.475 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-27T22:33:03.475 INFO:tasks.workunit.client.0.smithi042.stdout:19 2024-06-27T22:33:03.476 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:33:03.476 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:33:03.715 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:33:04.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:33:04.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:33:04.718 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-27T22:33:04.718 INFO:tasks.workunit.client.0.smithi042.stdout:20 2024-06-27T22:33:04.720 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:33:04.720 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:33:04.955 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:33:05.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:33:05.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:33:05.957 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-27T22:33:05.958 INFO:tasks.workunit.client.0.smithi042.stdout:21 2024-06-27T22:33:05.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:33:05.958 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:33:06.198 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:33:07.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:33:07.200 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:33:07.201 INFO:tasks.workunit.client.0.smithi042.stdout:22 2024-06-27T22:33:07.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-27T22:33:07.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:33:07.201 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:33:07.450 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-27T22:33:08.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-27T22:33:08.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-27T22:33:08.452 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-27T22:33:08.453 INFO:tasks.workunit.client.0.smithi042.stdout:23 2024-06-27T22:33:08.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-27T22:33:08.453 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-27T22:33:08.738 INFO:tasks.workunit.client.0.smithi042.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/4133026267,v1:127.0.0.1:6817/4133026267] [v2:127.0.0.1:6818/4133026267,v1:127.0.0.1:6819/4133026267] exists,up ca3bab8b-59b0-4848-bc26-76728d8c1e01 2024-06-27T22:33:08.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-27T22:33:08.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-27T22:33:08.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-27T22:33:08.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:127: TEST_check_version_health_2: run_mgr td/ver-health x 2024-06-27T22:33:08.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-06-27T22:33:08.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:33:08.739 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-27T22:33:08.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:33:08.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/x 2024-06-27T22:33:08.740 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:33:09.123 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:33:09.123 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:33:09.123 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:33:09.123 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:33:09.123 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:33:09.123 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:33:09.123 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:33:09.124 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:33:09.125 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/x '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:33:09.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:128: TEST_check_version_health_2: run_mgr td/ver-health y 2024-06-27T22:33:09.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-06-27T22:33:09.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-27T22:33:09.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=y 2024-06-27T22:33:09.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-27T22:33:09.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/y 2024-06-27T22:33:09.161 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-27T22:33:09.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-27T22:33:09.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:33:09.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:33:09.531 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:33:09.531 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:33:09.531 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:33:09.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:33:09.531 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:33:09.532 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/y '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-27T22:33:09.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:129: TEST_check_version_health_2: run_mds td/ver-health m 2024-06-27T22:33:09.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-06-27T22:33:09.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-06-27T22:33:09.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-06-27T22:33:09.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-06-27T22:33:09.567 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-06-27T22:33:09.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-06-27T22:33:09.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:33:09.567 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:33:09.567 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:33:09.568 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:33:09.568 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:33:09.569 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:33:09.569 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-06-27T22:33:09.592 INFO:tasks.workunit.client.0.smithi042.stdout:starting mds.m at 2024-06-27T22:33:09.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:130: TEST_check_version_health_2: run_mds td/ver-health n 2024-06-27T22:33:09.599 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-06-27T22:33:09.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-06-27T22:33:09.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=n 2024-06-27T22:33:09.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-06-27T22:33:09.600 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/n 2024-06-27T22:33:09.600 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-06-27T22:33:09.600 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:33:09.601 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:33:09.601 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:33:09.601 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:33:09.601 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:33:09.601 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:33:09.602 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id n --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/n '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-06-27T22:33:09.650 INFO:tasks.workunit.client.0.smithi042.stdout:starting mds.n at 2024-06-27T22:33:09.655 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:132: TEST_check_version_health_2: sleep 5 2024-06-27T22:33:14.656 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:133: TEST_check_version_health_2: ceph health detail 2024-06-27T22:33:15.295 INFO:tasks.workunit.client.0.smithi042.stdout:HEALTH_OK 2024-06-27T22:33:15.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: ceph health detail 2024-06-27T22:33:15.313 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: grep DAEMON_OLD_VERSION 2024-06-27T22:33:15.989 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:137: TEST_check_version_health_2: kill_daemons td/ver-health KILL mon.b 2024-06-27T22:33:15.990 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:33:15.990 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:33:15.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:33:15.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:33:15.991 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:33:16.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:33:16.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-06-27T22:33:16.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: run_mon td/ver-health b --mon_warn_older_version_delay=0 2024-06-27T22:33:16.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-06-27T22:33:16.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-27T22:33:16.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-06-27T22:33:16.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-27T22:33:16.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-06-27T22:33:16.099 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --mon_warn_older_version_delay=0 2024-06-27T22:33:16.125 INFO:tasks.workunit.client.0.smithi042.stderr:2024-06-27T22:33:16.123+0000 7f0add26ec40 -1 'td/ver-health/b' already exists and is not empty: monitor may already exist 2024-06-27T22:33:16.128 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-27T22:33:16.128 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:33:16.128 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:33:16.129 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:33:16.129 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:33:16.130 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:33:16.130 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:33:16.130 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_warn_older_version_delay=0 2024-06-27T22:33:16.364 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-27T22:33:16.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-06-27T22:33:16.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:33:16.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T22:33:16.365 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-27T22:33:16.366 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-27T22:33:16.366 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T22:33:16.366 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T22:33:16.367 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T22:33:16.367 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:33:16.367 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:33:16.367 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:33:16.367 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.183441/ceph-mon.b.asok 2024-06-27T22:33:16.367 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:33:16.367 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.183441/ceph-mon.b.asok config get fsid 2024-06-27T22:33:16.485 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-06-27T22:33:16.485 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-27T22:33:16.485 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-06-27T22:33:16.486 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-27T22:33:16.487 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-27T22:33:16.487 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-06-27T22:33:16.487 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-06-27T22:33:16.487 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-06-27T22:33:16.487 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-27T22:33:16.487 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:33:16.487 INFO:tasks.workunit.client.0.smithi042.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:33:16.487 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.183441/ceph-mon.b.asok 2024-06-27T22:33:16.488 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-27T22:33:16.488 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.183441/ceph-mon.b.asok config get mon_host 2024-06-27T22:33:16.607 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:142: TEST_check_version_health_2: kill_daemons td/ver-health KILL mds.m 2024-06-27T22:33:16.608 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:33:16.608 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:33:16.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:33:16.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:33:16.608 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:33:16.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:33:16.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-06-27T22:33:16.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: run_mds td/ver-health m 2024-06-27T22:33:16.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-06-27T22:33:16.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-06-27T22:33:16.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-06-27T22:33:16.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-06-27T22:33:16.717 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-06-27T22:33:16.718 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-06-27T22:33:16.718 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-27T22:33:16.718 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-27T22:33:16.718 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-27T22:33:16.718 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:33:16.718 INFO:tasks.workunit.client.0.smithi042.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:33:16.718 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.183441/$cluster-$name.asok' 2024-06-27T22:33:16.719 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.183441/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-06-27T22:33:16.954 INFO:tasks.workunit.client.0.smithi042.stdout:starting mds.m at 2024-06-27T22:33:16.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:145: TEST_check_version_health_2: wait_for_health_string 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-06-27T22:33:16.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There are daemons running an older version of ceph' 2024-06-27T22:33:16.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-06-27T22:33:16.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-06-27T22:33:16.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-06-27T22:33:16.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-06-27T22:33:16.960 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-27T22:33:16.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-27T22:33:16.961 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-06-27T22:33:17.551 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-06-27T22:33:18.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-06-27T22:33:18.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-06-27T22:33:18.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-06-27T22:33:18.553 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-06-27T22:33:19.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-06-27T22:33:19.140 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-06-27T22:33:19.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-06-27T22:33:19.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-06-27T22:33:19.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-06-27T22:33:19.141 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:147: TEST_check_version_health_2: ceph health detail 2024-06-27T22:33:19.836 INFO:tasks.workunit.client.0.smithi042.stdout:HEALTH_WARN There are daemons running an older version of ceph 2024-06-27T22:33:19.836 INFO:tasks.workunit.client.0.smithi042.stdout:[WRN] DAEMON_OLD_VERSION: There are daemons running an older version of ceph 2024-06-27T22:33:19.836 INFO:tasks.workunit.client.0.smithi042.stdout: mon.b mds. are running an older version of ceph: 01.00.00-gversion-test 2024-06-27T22:33:19.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: ceph health detail 2024-06-27T22:33:19.855 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-06-27T22:33:20.597 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: ceph health detail 2024-06-27T22:33:20.598 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There are daemons running an older version of ceph' 2024-06-27T22:33:21.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: ceph health detail 2024-06-27T22:33:21.263 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: grep -q 'mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test' 2024-06-27T22:33:21.849 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: return 1 2024-06-27T22:33:21.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: return 1 2024-06-27T22:33:21.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2332: main: code=1 2024-06-27T22:33:21.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/ver-health 1 2024-06-27T22:33:21.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-06-27T22:33:21.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-06-27T22:33:21.850 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-06-27T22:33:21.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-27T22:33:21.851 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-27T22:33:21.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-27T22:33:21.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-27T22:33:21.851 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-27T22:33:22.166 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-27T22:33:22.166 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-27T22:33:22.167 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-27T22:33:22.168 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-27T22:33:22.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-27T22:33:22.169 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-27T22:33:22.170 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-27T22:33:22.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:33:22.171 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-27T22:33:22.172 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-27T22:33:22.172 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T22:33:22.173 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:33:22.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-06-27T22:33:22.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-06-27T22:33:22.174 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-06-27T22:33:22.176 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/ver-health/mds.m.log td/ver-health/mds.n.log td/ver-health/mgr.x.log td/ver-health/mgr.y.log td/ver-health/mon.a.log td/ver-health/mon.b.log td/ver-health/osd.0.log td/ver-health/osd.1.log td/ver-health/osd.2.log /home/ubuntu/cephtest/archive/log 2024-06-27T22:33:22.177 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-06-27T22:33:22.190 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-27T22:33:22.190 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-27T22:33:22.190 INFO:tasks.workunit.client.0.smithi042.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.183441 2024-06-27T22:33:22.190 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.183441 2024-06-27T22:33:22.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-27T22:33:22.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-27T22:33:22.192 INFO:tasks.workunit.client.0.smithi042.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 1 2024-06-27T22:33:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T22:33:22.193 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-06-27T22:33:22.193 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-06-27T22:33:22.547 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/ver-health.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' 2024-06-27T22:33:22.727 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=51ef79b8d1cc4d7fbcbaf3760c16662c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/ver-health.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' 2024-06-27T22:33:22.730 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-27T22:33:22.739 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-27T22:33:22.739 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-27T22:33:22.775 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-27T22:33:22.784 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-27T22:33:22.784 DEBUG:teuthology.orchestra.run.smithi042:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-27T22:33:22.815 INFO:teuthology.orchestra.run.smithi042.stdout: remote refid st t when poll reach delay offset jitter 2024-06-27T22:33:22.816 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================== 2024-06-27T22:33:22.816 INFO:teuthology.orchestra.run.smithi042.stdout:+hv01.front.sepi 67.205.162.81 3 u 16 64 377 0.067 -0.862 0.117 2024-06-27T22:33:22.816 INFO:teuthology.orchestra.run.smithi042.stdout:*hv02.front.sepi 63.231.80.2 3 u 4 64 377 0.094 +0.885 0.086 2024-06-27T22:33:22.816 INFO:teuthology.orchestra.run.smithi042.stdout:+hv03.front.sepi 74.6.168.72 3 u 18 64 377 0.133 +0.357 0.027 2024-06-27T22:33:22.816 INFO:teuthology.orchestra.run.smithi042.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-06-27T22:33:22.817 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-27T22:33:22.826 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-27T22:33:22.826 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-27T22:33:22.834 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-27T22:33:22.841 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-27T22:33:22.849 INFO:teuthology.task.internal:Duration was 4253.924887 seconds 2024-06-27T22:33:22.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-27T22:33:22.857 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-27T22:33:22.858 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-27T22:33:22.903 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-27T22:33:22.903 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com 2024-06-27T22:33:22.903 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-27T22:33:22.961 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-27T22:33:22.961 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-27T22:33:23.115 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-27T22:33:23.115 DEBUG:teuthology.orchestra.run.smithi042:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-27T22:33:23.178 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-27T22:33:23.187 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-27T22:33:23.187 DEBUG:teuthology.orchestra.run.smithi042:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-27T22:33:23.201 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-27T22:33:23.210 DEBUG:teuthology.orchestra.run.smithi042:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:33:23.252 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = core 2024-06-27T22:33:23.282 DEBUG:teuthology.orchestra.run.smithi042:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:33:23.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T22:33:23.328 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-27T22:33:23.338 INFO:teuthology.task.internal:Transferring archived files... 2024-06-27T22:33:23.340 DEBUG:teuthology.misc:Transferring archived files from smithi042:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775602/remote/smithi042 2024-06-27T22:33:23.341 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-27T22:33:24.177 INFO:teuthology.task.internal:Removing archive directory... 2024-06-27T22:33:24.177 DEBUG:teuthology.orchestra.run.smithi042:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-27T22:33:24.218 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-27T22:33:24.228 INFO:teuthology.task.internal:Not uploading archives. 2024-06-27T22:33:24.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-27T22:33:24.239 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-27T22:33:24.239 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-27T22:33:24.264 INFO:teuthology.orchestra.run.smithi042.stdout: 658466 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 27 22:33 /home/ubuntu/cephtest 2024-06-27T22:33:24.264 INFO:teuthology.orchestra.run.smithi042.stdout: 673927 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 27 21:31 /home/ubuntu/cephtest/mnt.0 2024-06-27T22:33:24.264 INFO:teuthology.orchestra.run.smithi042.stdout: 675954 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 27 22:28 /home/ubuntu/cephtest/mnt.0/client.0 2024-06-27T22:33:24.264 INFO:teuthology.orchestra.run.smithi042.stdout: 791317 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 27 22:28 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-27T22:33:24.265 INFO:teuthology.orchestra.run.smithi042.stdout: 791318 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Jun 27 22:33 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-06-27T22:33:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T22:33:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-06-27T22:33:24.266 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi042 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-27T22:33:24.270 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-27T22:33:24.281 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-27T22:33:24.293 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi042 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-27T22:33:24.297 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} duration: 4253.924887180328 failure_reason: 'Command failed (workunit test misc/ver-health.sh) on smithi042 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=51ef79b8d1cc4d7fbcbaf3760c16662c status: fail success: false 2024-06-27T22:33:24.297 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-27T22:33:24.388 INFO:teuthology.run:FAIL